AI Agent Observability: The Trace That Lied About Success

The root span said OK. One child tool span said ERROR. A retry succeeded, the user received an answer, and a dashboard that grouped by root status would have counted the run as clean. That is a believable trace and a misleading incident record.

That is the observability lie.

I put that five-span record through a local export-admission probe, then changed one field at a time. The recovered baseline was allowed to export because its root explicitly reconciled one child error and the retry linked back to the failed attempt. Twenty-seven hostile variants were blocked. Two valid variants reached EXPORT. All 29 expected decisions matched.

This is the useful boundary for AI agent observability with OpenTelemetry: instrumentation is not finished when spans exist. Before telemetry leaves the trust boundary, the trace still has to prove parentage, bounded identity, honest error accounting, safe attributes, and a sampling decision that does not discard the failure operators need to see.

The root span told only half the story

An agent run is not a normal request with extra labels. It can plan, call a model, invoke tools, retry a failed tool, hand work to another process, and still return a successful outer response. The trace must preserve those operations as related spans instead of compressing them into one “agent completed” event.

OpenTelemetry defines a span as a unit of work with a trace ID, span ID, optional parent, timestamps, attributes, events, links, and status. A root has no parent. Child spans inherit the trace ID and point to the span that owns their work. Links cover causal relationships that do not fit ordinary parentage, including a later retry or asynchronous continuation.

The GenAI semantic conventions now live in a dedicated OpenTelemetry repository. The agent conventions are still marked Development. They describe operations such as invoke_agent, plan, and execute_tool; the general GenAI span conventions cover inference and token usage. That status matters: pin the convention revision you implement, and review upgrades like schema migrations rather than assuming every backend will map a new attribute automatically.

What the synthetic trace actually contained

The baseline is deliberately small: one root agent span, one planning span, two tool spans, and one model span. The first tool attempt ends in ERROR with error.type=timeout. The second attempt succeeds and links to the failed span. The root reports agent.outcome=recovered and agent.recovered_error_count=1.

{
  "root": {
    "name": "invoke_agent support-agent",
    "status": "OK",
    "outcome": "recovered",
    "recovered_error_count": 1
  },
  "failed_tool": {
    "name": "execute_tool shipment_lookup",
    "status": "ERROR",
    "error.type": "timeout",
    "attempt": 1
  },
  "retry": {
    "status": "UNSET",
    "attempt": 2,
    "links": ["failed_tool_span_id"]
  },
  "tail_sample_decision": "KEEP"
}

The model span carries requested and response model families, 894 synthetic input tokens, 312 synthetic output tokens, a synthetic estimated cost of $0.0018, and a pricing-table version. None of those values came from a provider. They exist only to exercise type, provenance, and policy checks.

The fixture SHA-256 is a2f99086ed0386bbd4d5a65cc0e302df71d166af240e8e6afe75516c071cfea3. The dependency-free probe did not initialize an OpenTelemetry SDK, Collector, backend, model, agent, credential, or network exporter. It evaluated JSON in memory and wrote a hash-bound findings file.

Validation: 29 export decisions

The remaining mutations covered missing resource identity, mismatched trace IDs, open spans, negative time, a duration ceiling, span and attribute ceilings, a high-cardinality span name, an unknown operation name, a missing error.type, invalid token counts, cost without a pricing version, and disabled redaction. The probe returns the first applicable blocker so a failed export has one stable remediation target.

Parentage is the first audit trail

Trace assembly should fail closed before attribute quality becomes interesting. A trace with two roots is not “mostly connected.” A child whose parent does not exist is not harmless noise. A cycle cannot describe elapsed work. Duplicate span IDs make later joins unsafe even if the visual waterfall appears plausible.

Pretty waterfalls are not proof.

The local admission order reflects that priority:

1. require stable resource identity;
2. require exactly one root and one trace ID;
3. prove unique span IDs, existing parents, and acyclic ancestry;
4. prove every span ended with non-negative timing inside the trace budget;
5. then evaluate attributes, outcome reconciliation, retry links, and sampling.

This ordering also keeps alerts useful. A malformed parent graph can cause ten downstream anomalies. Reporting the broken graph first is more actionable than producing ten low-level warnings from data that should never have advanced.

Error recovery must reconcile upward

OpenTelemetry span status has three values: Unset, Error, and Ok. Unset normally represents a completed operation without an error; explicitly setting Ok is a final, unambiguous success judgment. That makes a root OK meaningful, not decorative.

A recovered agent run can legitimately end OK while a child span remains ERROR. The problem is failing to say that recovery occurred. The local contract requires the root to label its outcome recovered and to report the exact number of failed descendants. It also requires a later retry to link to a failed span.

Those two fields are local policy, not OpenTelemetry standard attributes. They are useful because they make the apparent contradiction testable. Another organization might represent recovery with a span event, a different namespace, or a derived backend field. What matters is that success queries cannot silently erase the error path.

Redaction belongs before vendor routing

The GenAI conventions mark inputs, outputs, system instructions, prompt variables, tool arguments, and tool results as opt-in content. “Opt-in” is not a suggestion to capture everything and clean it later. These fields can carry credentials, personal data, proprietary instructions, or an entire customer record.

The fixture rejects prompt and completion namespaces, tool arguments and results, authorization material, API-key fields, and end-user email fields. It also scans string values for a narrow set of secret-like patterns. That detector is intentionally incomplete. The stronger control is an allowlist of operational attributes built at instrumentation time, followed by a Collector transform or attribute processor before export.

OpenTelemetry documents the Collector as a place to filter, delete, insert, replace, or transform telemetry for governance, security, cost, and data quality. It also warns that advanced transformations can affect Collector performance. Treat redaction configuration as production code: test representative payloads, pin the configuration, measure processor pressure, and fail the pipeline when the policy cannot be loaded.

For a deeper redaction contract, see the companion AI agent trace redaction field note. It focuses on attribute-level export policy; this article focuses on whether the whole trace is internally honest enough to export.

Sampling has to see the whole failure

Head sampling decides near span creation, before the complete run exists. That is cheap and predictable, but the root cannot yet know that a later tool attempt will time out. Tail sampling evaluates accumulated trace data after spans arrive, which permits policies based on error status, latency, attributes, or span count.

The Collector Contrib tail-sampling processor groups spans by trace ID and requires all spans for a trace to reach the same Collector instance for effective decisions. Its documentation also calls out memory sizing, decision timing, caches for late spans, and a maximum trace-size control. Those are operating requirements, not optional polish.

The synthetic policy is intentionally blunt: any trace with an error span must end with KEEP. A real sampling program may combine error, latency, tenant, rate, and probabilistic policies. The useful regression case remains the same: inject a child error under a successful root and prove the trace survives.

Cost is computed policy, not telemetry truth

Token usage can be response data. Estimated currency cost is an application calculation. A provider can change pricing, apply a cached-token discount, route to a different model revision, or bill a tool separately. A naked cost_usd attribute is therefore impossible to audit.

The local trace admits an estimated cost only when a pricing-table version is present. In a production design, also record the currency, calculation timestamp or revision, and whether the number is estimated or reconciled. Keep the attribute low-cardinality and avoid embedding an entire rate card in each span.

The tradeoff is storage and query complexity. More provenance makes cost analysis defensible but expands every trace. If finance-grade reconciliation matters, traces should point to a versioned pricing record rather than trying to become the billing ledger.

Troubleshooting failures that deserve tickets

• Green root, invisible red child: fix outcome reconciliation and error-oriented sampling before tuning dashboards.
• Retry with no causal link: the second attempt may be visible, but the recovery story is not reproducible.
• Prompts in a convenience attribute: remove the field at instrumentation, then add Collector defense in depth.
• User data inside span names: cardinality and privacy failures are now baked into indexes and billing.
• Estimated cost without a rate revision: the number cannot survive a pricing change or audit.
• Tail sampler split across trace fragments: routing must ensure every span for a trace reaches the same decision point.
• Open spans during shutdown: flush and shutdown behavior needs a lifecycle test, not a sleep inserted before exit.

A release rule for agent traces

Volume is not evidence.

Do not release agent instrumentation because a trace appears in a backend. Release it when a deterministic fixture proves the graph, timing, bounded names, allowed attributes, error reconciliation, retry causality, and sampling outcome you intend to query during an incident.

The local gate used here is stricter than the OpenTelemetry data model in several places. That is deliberate. OpenTelemetry standardizes telemetry; it does not know which prompts are sensitive in your product, how many spans your Collector can afford, what constitutes a recovered run, or which pricing table makes an estimate defensible.

A GolemWorkers agent can centralize runtime configuration and scoped secrets, but observability still needs its own export contract. If the trace cannot tell the truth about one failed tool retry without leaking the user’s data, it is not production evidence yet.

Sources and revision notes

• OpenTelemetry traces: spans, parentage, links, attributes, status, and context
• OpenTelemetry GenAI agent and framework span conventions
• OpenTelemetry generative AI client span conventions
• OpenTelemetry Collector telemetry transformation guidance
• OpenTelemetry Collector Contrib tail-sampling processor