Most technology leaders report that their companies log all AI agent activity, including each model and tool call. However, retrieving a complete run from last quarter—including accessed data, modifications, authorizations and rationale—requires a series of critical decisions during configuration.
Few of these settings receive legal or risk review. To control costs, the collection layer retains only a random subset of runs. A single environment variable controls whether prompt text and tool results are sent to a vendor’s servers. The vendor plan sets trace retention, and the data format affects switching costs.
Companies generally learn of unanticipated agent actions only after the fact. A 2026 survey of cybersecurity professionals found that 91% of organizations discover agent activity retrospectively, while only 23% enforce AI security in real time. When oversight is retrospective, records serve as the main evidence that controls were effective. Record quality affects response times to regulators or legal inquiries and determines whether engineers can confirm their completeness.
Agent record-keeping requires leadership oversight. OpenTelemetry, a widely adopted data collection framework, now provides an open standard for recording agent activity, capturing each step and its key details. Businesses must define the scope of records in advance, set retention periods after sampling, and specify which actions require permanent records for compliance.
A Record Shaped Like the Work
An agent operates in a loop. It receives a goal, calls a model, selects a tool, reviews the result, and decides whether to continue. A single customer request can generate dozens of model calls and writes across internal services and external APIs. Engineers report that a typical run records 30 to 60 steps, and even more for longer runs.
Flat logs record each step separately and lose the connections between them. These links are essential because they show which tool result led to each agent action. Observability tools preserve these relationships in a trace structure where a tree with a single task root branches for each step. Each branch, or span, includes its own timing and status.
The trace structure is especially valuable for failures that do not trigger errors. For example, if a lookup returns nothing and the agent continues, each call still reports success, and error dashboards remain green. A trace allows engineers to query all empty results from the policy tool in the past 30 days, grouped by agent version. This task would take hours with flat logs.
The Standard Is Real and Still Moving
OpenTelemetry (OTel) established a generative AI working group in April 2024. Its conventions define agent system operations, from individual model calls to workflows involving multiple agents, and specify fields such as model used, token count, duration and tool invoked. Several observability vendors support the format directly, while others translate it upon receipt. Because platform and security teams already use OTel, adopting it for agents requires minimal additional training.
OTel rates each convention from Development to Stable, and all generative AI conventions are still at the Development stage. In June 2026, these conventions moved to a separate repository to allow faster updates. Field names have already changed. For example, the model provider field was renamed, and many libraries currently emit both versions during the transition.
An evolving standard introduces manageable risk. When a field name changes, dashboards or audit queries may break until updated. Teams reduce this risk by routing agent telemetry through a single internal wrapper per language, so renames require only one change, and by pinning library versions. The specification requires instrumentation to keep emitting the same convention version until teams choose to upgrade, letting them upgrade on their own schedule. An engineer should maintain the list of field names used in reports and audit queries and review release notes quarterly.
A proprietary vendor format gives the vendor control over exit costs. To mitigate this, include a contract clause requiring the vendor to ingest and export agent telemetry in OTel format and specify the supported convention version. This ensures you can switch vendors when necessary.
The Facts the Standard Leaves to You
The conventions describe agent actions in technical terms. Engineers, finance, auditors, and process owners all use the same record, but each group may need information the standard doesn't cover. Your team should add custom fields under a controlled prefix, with one person responsible for the schema. Consistent naming prevents cross-team query issues.
Release identity
Agent behavior changes when you update the prompt, model, agent code or tools. Most investigations compare behavior before and after these changes. Each step should include version information. The standard covers the model version, and teams should add the other version details at minimal cost.
Outcome
Finance requires outcome data to calculate cost per successful result, and process owners need it to confirm task completion. Since the standard lacks this field, add an outcome label at the end of each run. Most businesses use labels such as resolved, escalated, abandoned or failed.
The agent’s closing message cannot provide the outcome label, as it reflects only the agent’s perspective and may be inaccurate. The label should come from the actual system. After the run, a process checks the ledger or ticket against the goal and records the result using the run’s trace identifier. For example, if a refund does not reach the ledger, the run is marked as failed, even if the agent reported success to the customer.
Authority
Auditors need to know what the agent was authorized to do and whether it stayed within those limits, but tool call records only capture the action. Firms with strict agent governance assign each agent a mandate, which is a written grant specifying permitted actions, data, limits and required approvals. Each tool call record should include the mandate identifier, policy decision, agent identity and, if applicable, the approver’s name and approval time.
Denied and escalated calls should include the same fields, as blocked calls demonstrate that controls are effective. With these fields, auditors can retrieve all approvals, mandates and applicable limits for the previous quarter in a single query.
Judgment
Determining whether a step was correct requires a separate judgment from a reviewer or an automated check, which may itself be a model. The standard defines an evaluation result for this purpose, including a name, score, a label such as pass or fail, and a brief explanation, attached to the step it grades.
Checks that detect skipped identity steps or access to the wrong customer file require access to tool call arguments and results, such as refund amounts or customer numbers. These values are considered content under the latter policy and should be stored in a restricted tier, accessible only for necessary checks.
Following the Action Past the Agent
Customer-impacting changes often occur outside the agent, such as when a tool call interacts with a payments service or customer database. If the trace ends at the agent, it records only the request and omits the actual write.
Since February 2020, the W3C Trace Context recommendation has addressed this issue. It defines a short header, traceparent, which is attached to each request. Services with standard OTel instrumentation read this header and nest their steps under the agent’s step, creating a tree from the customer’s request to the database write. This linkage enables audits to connect the system of record’s log with the trace’s reasoning and authority.
Tools using the Model Context Protocol (MCP) now include the traceparent header in a metadata field on each request. OTel conventions establish this practice, and the protocol’s maintainers have formalized it in the specification, with several SDKs and gateways already compliant. Each tool server must read the header and record its own activity.
Trace continuity can break in predictable areas. Reviews of high-consequence agents should examine each potential break point.
Queues and batch jobs: If a tool places a message on a queue for later processing, the header must travel inside the message, or the downstream work starts a separate tree with no link to the agent.
External services: Card processors or vendor APIs may ignore your header. Record any returned reference, such as a payment or ticket number, in the tool step so the trace can reference the vendor’s record.
Human approvals: If a manager approves a step later in a separate application, this occurs outside the agent’s process. Record the approval identifier on the step that awaited approval.
Agents calling agents: If the trace context is dropped during a handoff, the task splits into separate trees, disconnecting the second agent’s actions from the original request.
Verifying trace continuity requires minimal effort. An engineer should execute an end-to-end task for each high-consequence agent and confirm that a single trace reaches the system of record’s write. Teams that skip this check often discover trace breaks during incidents, when they can't recover missing steps.
What Sampling Throws Away
Agent runs generate significantly more telemetry than standard web requests, making full retention impractical for most firms. The retention rule shapes all future investigations, yet few outside engineering review it.
The typical default samples runs at the start, such as retaining one in ten at random. For agents, this timing is ineffective. Early in a run, failures resemble successful runs, so random sampling often misses the runs most likely to require later investigation.
Tail sampling occurs after a run completes. OpenTelemetry’s collector retains all steps in memory until the run ends, then applies retention rules. Agent-specific rules retain runs with errors, denied tool calls, unusually long durations, or high costs. Newly released agent versions should be fully retained for the first few weeks, while a small random sample of other runs provides a baseline. Practitioners note that a small percentage of runs account for most token usage, so cost-based rules capture significant spending.
This approach increases infrastructure demands. By default, the collector waits 30 seconds before deciding on retention, but long-running agents may exceed this window. The collector may then evaluate incomplete runs, and late-arriving steps appear as new runs unless it tracks prior decisions, resulting in fragmented traces. All steps must reach the same collector, which requires a routing layer. If the buffer fills, the collector drops steps without notice. Drop counts should trigger alerts.
Timing introduces additional challenges. Outcome labels and evaluation results may arrive after the run, sometimes the next day, by which time the sampler has already made retention decisions based on incomplete information. For example, a refund that fails to reach the ledger may originate from a run that appeared routine and was discarded, leaving no trace for later review.
Firms address this gap in two main ways: retaining all traces in low-cost storage for several days before final deletion, and allowing late signals to trigger long-term retention, so failed outcomes or customer complaints preserve relevant runs. LangSmith, a popular agent tracing service, follows this model. It traces the last 14 days by default, and feedback or review queue placement extends retention to 400 days. Additionally, the collector calculates and stores metrics such as request counts, errors, tokens and costs before sampling, ensuring accurate reporting even after most traces are deleted.
Instrumentation introduces additional overhead. Recording full prompts and tool results increases memory and processing demands on the application host, so measure latency with content capture enabled and disabled. Instrumenting every small internal function adds unnecessary volume, so focus on standard-named operations and external system calls, which provide the most value.
What the Record May Contain
The most valuable debugging data is the actual text the model received and returned, plus each tool’s response. By default, conventions exclude this text and flag these fields as likely to contain personal data. An engineer can enable them with a single environment variable, but a test-environment debugging setting may remain active in production without notice. This can send customer prompts and regulated records to third-party systems, subject to their access controls and jurisdiction.
An effective policy defines rules for each data class and places the configuration switch under change control. Redaction should occur at the collector before data leaves your network, ensuring all platforms receive the same sanitized data stream.
Two Stores for Two Duties
Traces help engineers diagnose issues, but some obligations require different evidence. Firms relying on a single store for both purposes often discover shortcomings during audits or disputes. Article 12 requires high-risk AI systems to automatically record events over their lifetime, so providers and deployers can spot risks and monitor how the system runs.
Under Articles 19 and 26, providers and deployers must keep the logs under their control for at least six months, unless other Union or national law, particularly data protection law, says otherwise, and financial institutions must keep them in the documentation their sector rules already require. Regulation (EU) 2026/1744, in force since July, moved the start date for stand-alone high-risk systems to 2 December 2027 and left these duties in place.
The Act leaves open what an adequate log contains. The international logging standard, ISO/IEC 24970, reached its final draft stage in 2026, and its European counterpart was still in public review in June. Until both are final, each firm decides for itself what a complete log looks like, and a regulator judges that decision afterward.
A sampled trace store falls short of that duty. It keeps only a fraction of runs and expires the rest on a cost-based schedule. What survives depends on a sampling configuration that engineers can change without a change record, and anyone with the right permissions can edit or delete its contents. A regulator asking for proof that a record hasn't been altered will not accept such a store as evidence.
A practical approach feeds two stores from the same events. The trace store retains sampled traces for debugging and cost analysis, with short retention and policy-governed content. The evidence store maintains a complete record of every consequential action by a high-risk system or one regulators consider comparable, with each entry containing the following elements:
The trace and conversation identifiers
The agent and its version
The mandate and the policy decision
The inputs the decision relied on, as redacted values or references to the system of record
The outcome, once known
Any human approval, with who gave it and when
Determining what is consequential requires a rule for each agent. Actions are consequential if they move money, change customer entitlements, alter records relied on by other processes, or cannot be reversed without human intervention. Each agent’s mandate should specify which tools meet these criteria and assign responsibility to the business owner who approved the mandate.
The evidence store remains compact because each entry is a brief structured record. For example, an agent approving 10,000 payments daily would generate about 20 megabytes per day, or roughly seven gigabytes per year, which is inexpensive to retain long term on write-once storage.
The evidence store allows new entries but prohibits edits. For example, in Amazon S3’s compliance mode, no user, including the root account, can overwrite or delete a locked record before its retention date. US broker-dealer regulations accept multiple approaches. Since 2023, SEC Rule 17a-4 permits either write-once storage or a time-stamped audit trail that records all changes and allows the original to be recreated. Regardless of the method, those who access the store should be separate from those who administer it.
Data protection laws introduce additional requirements. The GDPR requires retaining only necessary personal data for the required duration, and the AI Act defers to data protection law on log-retention periods. The evidence store addresses this by storing references, not content, where possible. For example, a hashed customer number and a pointer to the claims system allow investigators to reconstruct cases without duplicating customer files.
Both stores use shared identifiers, enabling auditors to cross-reference decisions and traces. For example, if a regulator inquires about a disputed high-risk credit decision, the evidence store can quickly provide the relevant entry, including agent version, mandate, policy decision, and reviewer. If the trace store retained the run, engineers can review the full trace. Even after trace expiration, the evidence entry remains valid and unchanged.
For non-high-risk systems, maintaining an evidence store is a business decision. Firms whose agents handle financial transactions should consider building one for all consequential actions, regardless of legal classification.
Who Owns Each Part
Each component of this design involves a risk judgment and requires an owner outside the agent development team. The data standards owner should manage the business field schema, while process owners define outcome labels. Because the sampling policy determines which runs regulators can access, risk and legal teams must approve it and manage changes through standard change control. Enabling content capture in production should require security and privacy approval and trigger an alert. Legal counsel sets evidence store retention periods, balancing the AI Act, sector rules, and data protection limits. A platform team member should review OpenTelemetry release notes quarterly and inform dependent teams of changes. Engineering is responsible for implementation and operations, and all owners should be listed in the same register as agent mandates.
What to Put in the Contract
Once you set the standard and fields, selecting a platform becomes a defined process. Require each vendor to demonstrate the following capabilities using your own traces, not a prepared demonstration.
Native ingestion of the OTel format for generative AI, with the supported version of the conventions stated and nothing dropped in translation
A full tree view of a real run of 40 or more steps, with token and cost totals at each level
Sampling after the run ends, with rules on your own fields and a holding window for late signals
Late evaluation results and outcome labels attached to the right run, including runs already in cheaper storage
Redaction, field-level access control and separate retention periods for each class of data
Raw export of all your data in OTel format at any time, so leaving costs one migration
Queries on your own outcome and policy fields that run as fast as queries on the built-in ones
The last point is critical. If the platform treats your outcome and authority fields as secondary, teams will revert to exporting data to spreadsheets. The contract should specify data location, vendor access rights, whether data may be used for model training or tuning, and data deletion timelines. The platform will store detailed records of customer requests and agent actions, so contractual terms must ensure the same level of care as any other system of record.
Before the Question Arrives
Many firms invest in agent observability only after a costly incident, when they cannot recover missing records. Proactive action is inexpensive compared to agent operating costs, as most work involves configuration and a few key decisions with assigned owners. Implementation does not need to wait for the standard to reach Stable status.
A practical first step requires one meeting. Ask the platform team to present the complete trace for a consequential run from 30 days ago, including outcomes and mandates for each action, down to the system-of-record write. Then, ask which parts of the trace would have been retained if the run had appeared routine at completion.
Trusted AI is written for executives, board members and senior practitioners building, funding or overseeing AI programs. Have a topic you want covered? Reply to any issue or reach out on LinkedIn.
Sources
Tian Pan, Your Agent Traces Are Lying: Cardinality, Sampling, and Span Hierarchies for LLM Agents
Tian Pan, The OpenTelemetry Tail Sampler That Dropped Exactly the LLM Spans Your Post-Mortem Needed
TrueFoundry, OpenTelemetry GenAI Conventions: A Common Vocabulary for AI Observability
Greptime, How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools
OpenObserve, OpenTelemetry GenAI Semantic Conventions: A Practical Guide
Honeycomb, Instrumenting AI Agents: Agent Timeline with OpenTelemetry
New Relic, Send Your GenAI Traces to New Relic with OpenTelemetry
Model Context Protocol, SEP-414: Document OpenTelemetry Trace Context Propagation Conventions
LangChain, Why are my traces being automatically upgraded to extended retention?
European Union, Regulation (EU) 2024/1689 (Artificial Intelligence Act)
AI Act Explorer, Article 26: Obligations of Deployers of High-Risk AI Systems
White & Case, EU AI Omnibus Enters Into Force, Amending the AI Act
StandICT.eu, ISO/IEC FDIS 24970 Artificial Intelligence: AI System Logging
Help Net Security, What the EU AI Act Requires for AI Agent Logging
Dechert, SEC Adopts Amendments to Rule 17a-4 Electronic Recordkeeping Requirements




Jon, I would want to test how the evidence record is created, as well as whether it stays intact.
The distinction you draw between diagnostic traces and a durable record is useful. My concern is that a record can preserve everything it receives and still leave out something relevant.
In an isolated test environment, I would introduce known gaps or delayed outcomes and check whether the review record makes them visible.
What would you need to see before accepting that capture is complete enough for the decision under review, beyond showing that the stored entries have not been altered?