AI Agent Trace Redaction: Keep Secrets Out of Spans

A prompt-redaction switch is useful, but it is not a trace boundary. Agent content can also land in resource attributes, custom span fields, exception events, status messages, links, and log bodies. A production policy therefore needs three separate controls: minimize content in the agent runtime, enforce an export schema at the Collector, and scan the serialized export with a harmless canary before release.

That conclusion came from a synthetic trace fixture, not from a marketing diagram. We placed one canary value in nine positions across seven telemetry containers. Removing the familiar prompt, tool, and exception-detail fields reduced nine occurrences to six. Modeling the current OpenTelemetry redaction processor's documented and source-visible coverage reduced six to two. The survivors were a span status message and a span-link attribute. A complete export policy removed both while retaining service identity, environment, operation, model, token count, error type, status code, and log severity.

The canary escaped through the boring fields

The first version of the fixture was intentionally impolite. It put gw-canary-trace-redaction-7d31 in a model prompt, tool arguments, a custom customer note, a resource note, an exception message, the span status description, a link attribute, a log body, and an authorization-shaped log attribute. None of those values was a credential. The point was to give every leak the same unmistakable marker.

Two leaks are still leaks.

The interesting number is not zero. It is two. The current redaction processor source walks resource attributes, instrumentation-scope attributes, span attributes, span-event attributes, log attributes, and log bodies. Its trace path does not show equivalent handling for a span's status message or link attributes. That makes the processor a strong schema firewall, but not evidence that every byte in an exported trace has been examined.

Minimize content where the agent creates it

The OpenAI Agents SDK documents that generation spans may store model input and output, while function spans may store function input and output. It also documents that trace_include_sensitive_data defaults to True. For production runs, reverse that default in both deployment configuration and code review:

from agents import Agent, Runner, RunConfig

run_config = RunConfig(trace_include_sensitive_data=False)

result = await Runner.run(
    Agent(name="support-router", instructions="Route the request."),
    input=user_request,
    run_config=run_config,
)

The equivalent environment control is OPENAI_AGENTS_TRACE_INCLUDE_SENSITIVE_DATA=false. Voice pipelines need their own trace_include_sensitive_audio_data setting because audio spans can otherwise contain base64-encoded PCM. These switches are valuable precisely because they prevent payloads from entering the trace in the first place.

They do not govern every custom span, library event, log record, or application-generated error. Nor do they make a second trace processor automatically safe: the SDK distinguishes adding a processor from replacing its default processors. If data must go only to an approved backend, review the complete processor set and export path rather than assuming an additional sanitizer intercepts every destination.

Validation: use the Collector as a schema firewall

OpenTelemetry's security guidance starts with data minimization, then recommends Collector processors when the application cannot control every field. The redaction processor is the most direct fit for a default-deny attribute policy: with allow_all_keys: false, unlisted keys are removed, while blocked-value patterns can mask secrets that appear inside an otherwise approved field.

processors:
  redaction/agent_export:
    allow_all_keys: false
    allowed_keys:
      - service.name
      - deployment.environment.name
      - gen_ai.operation.name
      - gen_ai.request.model
      - gen_ai.response.model
      - gen_ai.usage.input_tokens
      - gen_ai.usage.output_tokens
      - exception.type
      - error.type
      - error.code
      - severity
    blocked_key_patterns:
      - ".*(authorization|password|secret|token|api[_-]?key).*"
    blocked_values:
      - "(?i)bearer\\s+[a-z0-9._~-]+"
      - "gw-canary-[a-z0-9-]+"
    summary: info

This is an example policy, not a drop-in universal allowlist. Attribute names must match the SDK and semantic-convention version you actually ship. The redaction processor is beta for traces and alpha for logs and metrics at the time of writing, so pin the Collector distribution and version. Validate configuration against that binary, store the image digest, and treat any new allowed field as a reviewed data-flow change.

The two containers that need an explicit decision

Our fixture's remaining status message is easy to avoid: export the stable status code and error type, not a concatenated exception string. The remaining link attribute is a design choice too. A link can preserve trace and span identifiers without carrying email, tenant, prompt, or document context. If an exporter or processor later adds support for those containers, keep the canary gate anyway; capability changes are another reason to rerun the test.

A denylist alone is the weaker direction. Secret formats evolve, and ordinary personal data rarely has a reliable prefix. An allowlist answers a narrower operational question: which fields have an approved reason to leave the worker? Blocked-key and blocked-value patterns remain useful as a second tripwire, especially on approved text fields, but they should not define the export contract.

Make the exporter prove the policy

A source-code review can confirm intent. It cannot confirm the bytes sent by a deployed exporter. The release test should create a fresh harmless canary, inject it into every risky container supported by the instrumentation, send the data through the real Collector pipeline, and scan the fully serialized capture. Run the same test once with controls removed; that failure proves the harness is looking at the correct sink.

from pathlib import Path

CANARY = "gw-canary-trace-redaction-7d31"

def assert_clean_export(capture: Path) -> None:
    raw = capture.read_text(encoding="utf-8")
    if CANARY in raw:
        raise AssertionError("canary reached the trace export")

    required = [
        '"service.name"',
        '"gen_ai.operation.name"',
        '"gen_ai.usage.input_tokens"',
        '"status.code"',
    ]
    missing = [field for field in required if field not in raw]
    if missing:
        raise AssertionError(f"safe telemetry disappeared: {missing}")

Scanning only span attributes is not enough. Include resource and scope attributes, span names, events, status descriptions, links, logs, metric datapoints, and vendor envelopes. Test successes and failures. A tool exception often carries more sensitive context than a successful model call, and an exporter can serialize a field differently from the in-memory object inspected by a unit test.

A safe trace should still answer operator questions

Redaction fails organizationally when engineers lose the ability to diagnose incidents. The answer is not a larger payload; it is a deliberate set of low-content fields. In the fixture, the complete policy retained eight values: service name, deployment environment, operation name, request model, input-token count, exception type, status code, and log severity.

Correlation identifiers deserve special care. Plain hashing does not anonymize a small, predictable ID space; OpenTelemetry's guidance explicitly warns that those hashes can be reversible in practice. If correlation is necessary, prefer a keyed construction with a protected key, a documented rotation window, and a clear reason for cross-period linkage. Often a short-lived random run identifier is enough.

Diagnostic mode is a separate data product

Occasionally an incident needs content that the normal trace contract excludes. Do not turn that exception into a permanent global flag. Route a narrow worker cohort to an isolated backend, use synthetic or explicitly approved data, enforce a short expiry, record the operator and incident, and delete the additional data on schedule. Production traces and diagnostic captures should have different access, retention, and audit rules.

The same separation applies to logs. A trace identifier may connect a safe span to a restricted diagnostic record, but that does not make the restricted record safe for the trace vendor. Trace redaction is one boundary among several: runtime tool permissions, secret scoping, network egress, storage retention, and sandboxing still determine what the agent can reach before telemetry exists.

Failure modes the fixture cannot rule out

The measured result is narrow and reproducible. Nine synthetic canary placements became six after source minimization, two after modeling the current redaction processor's covered containers, and zero after status and link policy were added. Eight approved fields survived. No prompt, credential, customer record, production trace, or external exporter was read.

The test does not establish compatibility with a particular Collector release, OpenAI Agents SDK behavior, metrics redaction, exporter serialization, sampling, concurrency, or production latency. Those claims require the deployment-specific CI run described above. It also does not prove compliance with any law or standard; data classification, consent, residency, and retention remain the operator's responsibility.

For the surrounding telemetry model, continue with AI agent observability with OpenTelemetry. If untrusted tools are the larger risk, pair the export boundary with gVisor sandbox security. A GolemWorkers agent can centralize runtime configuration and scoped secrets, but it still needs this explicit trace policy before production data reaches an observability backend.

Sources

• OpenAI Agents SDK tracing — default tracing, sensitive generation/function content, audio content controls, and custom processor behavior.
• OpenTelemetry: handling sensitive data — operator responsibility, minimization, hashing limits, and Collector processor options.
• OpenTelemetry Collector redaction processor — allowlist semantics, blocked patterns, signal stability, configuration, and processor scope.
• Redaction processor implementation — current traversal of resource, scope, span, event, and log data used to define the fixture's modeled coverage.
• OWASP Logging Cheat Sheet — sensitive values that should not be recorded directly and the need to sanitize event data.