How rules run
A rule has four parts: the spans it looks at (its target), when it runs (its stage), the conditions a span must match, and the actions to take. In the editor it reads as a sentence: “When [any span] is [completed] and matches…, then…”.
Targets
Section titled “Targets”| Target | Looks at |
|---|---|
| any span | every span |
| a root span | spans with no parent in this process: a trace’s true root, or the entry span of a request that came from another service |
| a child span | spans whose parent is in this process |
“Root” is always local to the process. When service A calls service B, B’s server span is a root as far as B’s rules are concerned.
Stages
Section titled “Stages”| Stage | Runs | Can see |
|---|---|---|
| created | when the span starts | its name, start attributes, service and library, whether it’s a root, the current time |
| completed | when the span ends, and again when its trace’s root ends | everything above, plus end attributes, Span Duration and Has Error; the trace conditions only in the root-end pass |
Use completed for anything only known at the end. HTTP instrumentation, for example, sets http.route
and renames its span only as the span ends.
The trace conditions (Trace Duration, Trace Span Count, Spans In Trace) describe the whole trace, so they’re only known once the root ends. The editor offers them on completed rules.
Conditions
Section titled “Conditions”Conditions sit in groups joined by and or or, and groups can nest. A rule with no conditions matches every span its target covers.
| Condition | Operators |
|---|---|
| Service Name, Span Name, Library Name | is, is not, contains, does not contain, starts with, matches regex |
| Span Attribute | the text operators, plus exists, does not exist, =, >, >=, <, <=, between |
| Span Attribute Key | is, contains, starts with, matches regex, against every attribute key of the span |
| Span Duration, Trace Duration | =, >, >=, <, <=, between |
| Has Error | is true, is false |
| Trace Span Count | =, >, >=, <, <= |
| Spans In Trace | any, at least n, at most n, none, all, of the spans matching a nested condition |
| Current Time | between two times of day, in the process’s local time zone |
Library Name is the span’s instrumentation scope, such as @opentelemetry/instrumentation-knex.
Span Attribute Key checks every key of every span it reaches, so it’s the most expensive condition. SpanSlice evaluates it after the other conditions in its group; narrow the rule by span name or library first so it only runs on the spans that need it.
Rule order
Section titled “Rule order”Rules run in order: the service’s own rules first, then the project’s. Every matching rule runs its actions; a later rule can override what an earlier one set (see sampling for how rates combine). Stop Rule Evaluation ends rule processing for that span at that stage.
Spans are held until the trace is decided
Section titled “Spans are held until the trace is decided”The SDK holds every span of a trace until the trace’s local root ends. That’s what lets a completed rule on the root see the whole trace, and lets sampling keep or drop a trace as a unit.
To bound memory, a trace is sent early when it reaches 5,000 spans, is 120 seconds old, or more than 1,000 traces are open. A trace sent early skips the root-end pass and uses whatever rate its created rules set. Spans that end after their trace was sent follow the trace’s decision for 30 seconds.
Seeing which rule did what
Section titled “Seeing which rule did what”The SDK stamps the rules it applied on each span:
| Attribute | Meaning |
|---|---|
spanslice.rules / spanslice.rule_names |
every rule that matched the span |
spanslice.dropped_by / spanslice.dropped_by_name |
the rule whose Drop Span or Collapse Span removed it |
spanslice.sampled_by / spanslice.sampled_by_name |
the rule that set the trace’s sample rate or dropped the trace, on every span of the trace |