Sampling

SpanSlice has two sampling actions. They work at different levels:

Set Sample Rate decides whether the whole trace is kept.
Set Span Sample Rate thins out individual spans inside a trace that was kept.

	Set Sample Rate	Set Span Sample Rate
Decides	whether the trace is kept	whether this span is kept, inside a kept trace
Kept 1 in N by	trace ID, so every span of a trace agrees	span ID, so each span is decided on its own
What a “no” removes	the whole trace	the span; its children move up to the nearest kept ancestor
`SampleRate` on what’s sent	the trace rate	trace rate × span rate
Inside For Descendant Spans	no	yes
Settled	in rule order: created rules on the root, then completed rules	when the trace is sent

Set Sample Rate: keep 1 in N traces

“Keep 1 in 10 traces” keeps a tenth of the traces the rule matches and drops the rest, all spans together. Every span of a kept trace carries SampleRate = 10. Honeycomb reads that attribute and counts each event as ten, so counts, sums and rates in your queries still reflect the real traffic.

The keep-or-drop decision is computed from the trace ID. Every service using SpanSlice with the same rate makes the same decision for the same trace, so a sampled trace isn’t left with holes where one service kept its part and another didn’t.

Set Span Sample Rate: keep 1 in N of a span

Some traces are worth keeping but contain hundreds of near-identical spans: a Redis HSCAN loop, a batch of inserts, a polling call. Set Span Sample Rate keeps 1 in N of those spans and leaves the rest of the trace alone.

A span that isn’t kept is dropped, and its children are moved up to its nearest kept ancestor, so the trace stays connected.
A kept span carries SampleRate = trace rate × span rate. In a trace kept 1 in 10, a span kept 1 in 5 carries SampleRate = 50, because it stands for fifty such spans.
It’s decided when the trace is sent. A span still open at that point keeps the rate it had.

Because it acts per span, it can go inside For Descendant Spans. For example, on root spans:

When a root span is completed and matches Span Name is GET /reports, then For Descendant Spans matching Span Name starts with HSCAN: Set Span Sample Rate 1 in 20.

To thin out a span wherever it appears, use it directly in a rule targeting it instead:

When any span is completed and matches Span Name starts with HSCAN, then Set Span Sample Rate 1 in 20.

If you want the matching spans gone rather than sampled, use Drop Span. If you want them summarised, use Roll Up Spans, which replaces a run of like spans with one span carrying their count and timings.

When more than one rule sets a rate

Several rules can match the same trace. Both actions have a mode that says what happens when a rate is already set:

If Not Set (the default) applies only if no earlier rule set a rate.
Overriding Any Existing Sampling replaces whatever an earlier rule set.

“Earlier” follows one order: the created rules on the trace’s root first, then the completed rules, span by span in the order the spans started. Within a stage, rules run in rule order.

So with the default mode the first rate set wins. A broad rule such as “keep 1 in 10 traces” placed after specific ones acts as a fallback; a specific rule that must win regardless (say, “keep every trace from the billing service”) should be set to override.

Drop Trace and Drop Non-Errors

Drop Trace drops the trace outright and beats every rate, including overriding ones. A Drop Trace from a created rule sticks.
Drop Non-Errors drops the trace unless any span in it has an error.

Traces sent early

A trace sent before its root ends (it grew past the span or age limit, or the process shut down) skips the completed rules on the root and is kept at the rate its created rules set. See spans are held until the trace is decided.