Stuck spans
A span is only sent once it ends. A query waiting on a lock or a call to a service that never answers can stay open for minutes, and one that never ends is never seen. Stuck Span Detection reports a span while it’s still open, once it has been open longer than a Stuck Span Timeout.
Timeouts and handlers
Section titled “Timeouts and handlers”- A Stuck Span Timeout is how long a span can stay open before it counts as stuck, in seconds or minutes. New projects have none, so nothing is checked until you add one.
- Each timeout has an ordered list of handlers: conditions, and what to do with a span that matches. A handler with no conditions matches every span stuck that long.
A stuck span hasn’t ended, so its duration, error status and end attributes aren’t known yet. Handlers can test Span Name and Span Attribute only, with is or matches regex.
Like rules, timeouts are set per service and as project defaults, and switched on per environment. The service’s handlers are tried before the project’s.
What a handler does
Section titled “What a handler does”- Report once & drop (the default): one snapshot at the timeout. The span is dropped when it ends.
- Report then… or Report after… repeating N times, then…: one snapshot, or N, one each timeout,
then:
- Continue reporting indefinitely: a snapshot each timeout for as long as the span is open. The span is kept.
- Continue reporting N more times: N more snapshots. The span is dropped when it ends.
- Stop reporting & drop: no more snapshots. The span is dropped when it ends.
- Silently drop: no snapshot. The span is dropped when it ends.
A dropped span’s children move up to its parent, as with Drop Span. Open spans are checked every 5 seconds, so a snapshot can come up to 5 seconds after its timeout.
Snapshots
Section titled “Snapshots”A snapshot is a copy of the span as it looks right now. It’s named <span name> (incomplete), ends at
the moment it’s taken, and has its own span ID, so it never collides with the real span. It carries the
span’s attributes, with redactions applied, plus:
| Attribute | Meaning |
|---|---|
spanslice.stuck.is_snapshot |
true |
spanslice.stuck.duration_ms |
how long the span had been open |
spanslice.stuck.reported_count |
1 on the first snapshot, 2 on the next, … |
spanslice.stuck.source_span_id |
the real span’s ID, to find it once it ends |
spanslice.stuck_by |
the handler that took the span; the real span carries it too |
Snapshots are sent as they’re taken, outside the trace: rules and sampling don’t apply to them, and they
carry no SampleRate. In Live Tail, the Stuck filter shows them.
Several timeouts
Section titled “Several timeouts”Timeouts are checked from shortest to longest. At each one the span passes, the handlers are tried in order and the first match takes the span over from any earlier timeout’s handler, counting its own snapshots from one. No match leaves the earlier handler in charge; a span no handler matches is kept and never reported. The handler in charge reports once per its own timeout.
For example, with a 30-second timeout whose handler reports pg.query indefinitely, and a 5-minute one
that reports any span once and drops it, a query open for six minutes is reported at 30 seconds, every 30
seconds up to 4½ minutes, then once more at 5 minutes by the second handler, and dropped when it ends.
A trace with a stuck span open is held like any other, and sent early once it’s 120 seconds old.