Sampling live events without lying

A live board that shows every event will melt a consumer, a bill, or both. So teams sample. The lie begins when someone treats “12% of the socket” as “the funnel divided by ten, roughly.” Conversion is a ratio. Sampling that is not uniform across steps invents a ratio that never happened.

In the Live Usage Lab we use a Bangkok marketplace trace with three steps that actually exist: view_item, add_to_cart, checkout_paid. On a Tuesday peak, view_item is chatty and cheap. checkout_paid is rare and often retried. If you sample by “keep 12% of all messages,” paid events — being fewer — disappear faster in absolute counts, but retries keep them in the stream longer. The sampled conversion chip climbs. Product celebrates. Finance does not recognise the day.

Three checks before you trust the chip

First: sample by session key, not by message. If a session is in, all of its steps are in. If it is out, none of them are. This is slower to implement on a socket fan-out than a coin flip per event, which is why people avoid it. It is also the only way the ratio stays honest when step volumes differ.

Second: write the expected bias in the same sentence as the number. “Live conversion on a 12% session sample, retries de-duplicated by payment_id, PromptPay callbacks excluded until confirmed.” If that sentence is too long for the chip, the chip is too small, not the sentence.

Third: compare one hour of sampled live against the warehouse land for that same hour, once it arrives. Not to make the live board “match BI,” but to see whether sampling or lag is the cousin you are looking at. In Pomprab we keep a paper note of the delta. If the delta flips sign when it rains, you do not have a conversion story; you have a network story.

What we refuse to do

We refuse to “correct” a sampled live chip with a multiplier taken from last week’s warehouse. Last week was not this hour. Multipliers launder bias into confidence. If you need a trustworthy daily conversion, wait for the land or instrument a full-fidelity paid path and sample only the chatty views.

The lab worksheet asks pairs to compute conversion three ways on the same trace: per-message sample, per-session sample, and full fidelity. Nobody has finished that sheet believing the first number. That is the point. If you want to sit the exercise with your own path, read the Live Usage Lab syllabus or write the desk.