Your AI is Running. But is It Working?

The observability stack that makes AI systems debuggable, auditable, and improvable

You can monitor a traditional web application with dashboards that track error rates, latency percentiles, and throughput. When something breaks, you read the logs, find the stack trace, and fix the bug. The same input produces the same output. The system is deterministic. Debugging is hard, but the problem space is well understood.

AI systems break all of these assumptions.

The same prompt can produce different outputs on consecutive runs. A model that performed well last month may silently degrade as the data it encounters in production diverges from its training distribution. An agent that chains together retrieval, tool calls, and generation steps can fail at any point in the sequence, and the failure may not even look like an error. It may look like a confident, well-structured response that happens to be wrong.

This is why traditional application monitoring is necessary but insufficient for AI systems. You still need infrastructure metrics, error tracking, and latency measurement. But you also need a fundamentally different observability layer, one that captures the semantic content of AI interactions, the quality of outputs, the provenance of context, and the decisions made along the way.

This post examines what that observability stack looks like in practice. What to log, how to structure telemetry, how to capture feedback and evaluation signals, and how to do all of this without turning your logging pipeline into a privacy and compliance liability.

Why AI Observability Is Different

To understand why AI systems demand a different approach to observability, consider the specific properties that distinguish them from traditional software.

Non-deterministic outputs

Even with the temperature set to zero, model outputs can vary across providers, versions, and infrastructure configurations. Two requests with identical prompts may return substantively different responses. This means you cannot rely on simple assertions or on expected output matching what you would with a conventional API. Quality evaluation requires probabilistic assessment, not binary pass/fail checks.

Prompt sensitivity

Small changes to a prompt, sometimes a single word, can produce dramatically different results. A system prompt modification intended to improve formatting may inadvertently degrade factual accuracy. Without logging prompt versions alongside outputs, teams cannot correlate quality regressions to the changes that caused them.

Model and data drift

AI models do not remain static after deployment. Foundation models get updated by their providers. Fine-tuned models degrade when the distribution of production data shifts away from that of the training data. Retrieval-augmented generation systems depend on the quality and freshness of their knowledge bases, which are continuously updated. Traditional monitoring will tell you the system is running. It will not tell you the system is returning worse answers than it was a month ago.

Tool use and multi-step reasoning

Modern AI architectures increasingly involve agents that select and execute tools, retrieve context from external sources, and chain multiple inference calls together before producing a final response. A failure anywhere in this chain can compromise the output, but the failure may be invisible if you only inspect the final response. An agent might retrieve irrelevant documents, select the wrong tool, or misinterpret the tool's output, yet still produce a fluent, confident answer.

Harder debugging

When a traditional application returns an incorrect result, you can typically trace the logic path through deterministic code. When an AI system returns a hallucinated answer, the “bug” may reside in the prompt, the retrieved context, the model weights, the system instructions, or the interaction between all of these. Reproducing the issue often requires the exact combination of inputs, context, and model state that produced the original output.

These properties create a fundamental challenge. Standard logging tells you what happened at the infrastructure level. AI observability must tell you what happened at the semantic level, capturing enough information to understand not just whether the system responded, but whether it responded well.

According to Grafana’s 2026 Observability Survey, adoption of LLM-specific observability is accelerating but remains early. The percentage of organizations with no LLM observability on their radar dropped from 42% in 2025 to 29% in 2026, but only 14% are using it for production workloads. The gap between AI deployment and AI observability represents one of the most significant operational risks in enterprise technology today.

What to Log

The first question any AI observability program must answer is what to capture. Log too little, and you cannot debug problems or measure quality. Log too much, and you create performance overhead, storage costs, and privacy exposure. The right approach captures enough to reproduce behavior and assess quality without recording unnecessary raw content.

Here are the core event types that a production AI observability system should capture.

Prompts and system instructions

Every request to an AI model begins with some combination of system instructions, user input, and conversation history. Logging these inputs, or at minimum their hashed versions and associated version identifiers, is essential for debugging and for understanding how prompt changes affect output quality. The most practical approach is to log a prompt template version identifier rather than the full prompt text, then maintain a separate versioned prompt registry that allows reconstruction when needed.

Retrieved context

For retrieval-augmented generation systems, the documents or passages retrieved from vector stores or search indices are a critical determinant of output quality. Log the document identifiers, relevance scores, and retrieval metadata. If the model produces a hallucinated answer, you need to determine whether the retrieval step failed to surface the right information or whether the model ignored the relevant context it was given. Logging full document content is generally unnecessary and raises data governance concerns. Reference identifiers with scores are usually sufficient.

Model outputs

The model's response is the primary artifact of interest. For debugging and evaluation, teams need access to outputs. However, logging full output text in all cases can be expensive and may capture sensitive information. Many organizations implement tiered logging, capturing full outputs for a configurable sample of requests while logging only metadata such as output length, finish reason, and token counts for the remainder.

Tool calls and function invocations

When an AI system invokes tools or external functions, log the tool name, input parameters, return values, execution duration, and any errors. For agentic systems that make multiple tool calls in sequence, capture the ordering and dependencies among them. This is where observability for AI most closely resembles distributed tracing in microservices architecture, and it benefits from the same structural patterns.

User actions and session context

What did the user do after receiving the AI response? Did they accept a suggestion, edit it, reject it, escalate to a human, or abandon the session? These downstream actions are among the most valuable signals for measuring real-world quality, yet they are frequently overlooked. Log user interactions with sufficient context to correlate them back to the AI response that triggered them.

Latency and performance metrics

Time to first token, total response time, retrieval latency, and individual tool call durations all matter for user experience and cost management. Break latency into its component parts to identify bottlenecks. The retrieval step might cause a slow response, the model inference, or a downstream tool call, and the remedy differs in each case.

Token usage and cost

Token consumption directly translates to cost for API-based model deployments. Log input tokens, output tokens, and reasoning tokens separately for each request. Aggregate these by use case, user segment, prompt version, and model to understand cost drivers and identify optimization opportunities. Organizations running AI at scale frequently discover that a small number of use cases or prompt patterns account for a disproportionate share of their token spend.

Errors and exceptions

Beyond standard HTTP errors and timeouts, AI systems produce a category of failures that do not register as errors in traditional monitoring. A model that returns a refusal, a safety filter activation, a truncated response due to context window limits, or a response that fails a validation check should be logged as a structured event with sufficient context to diagnose the cause.

Metadata

Every logged event should carry standard metadata, including the model name and version, provider, environment, deployment identifier, and timestamp. This metadata enables filtering, correlation, and comparison across model versions, environments, and time periods.

The guiding principle is to log enough to reproduce and evaluate any interaction without capturing unnecessary raw content. For most organizations, this means a combination of structured metadata for every request, sampled full-content logging for a subset of requests, and on-demand full-content capture that can be activated for specific users, sessions, or time windows when debugging requires it.

How to Structure Telemetry

Knowing what to log is only half the challenge. How you structure that telemetry determines whether it is searchable, correlated, and actionable, or whether it becomes an expensive pile of unstructured text that no one can use effectively.

The dominant paradigm for structuring distributed system telemetry combines traces, spans, and events. This approach has been validated at scale in traditional observability and is now being adapted for AI workloads. The OpenTelemetry project has introduced experimental semantic conventions specifically for generative AI, defining standardized attributes for model calls, tool invocations, agent operations, and evaluation events. These conventions are emerging as the industry standard for AI telemetry.

Traces represent end-to-end interactions. A single user request that triggers retrieval, model inference, tool execution, and response generation should be captured as a single trace that encompasses the entire chain. Every trace should carry a unique trace ID, a session ID that groups related interactions within a conversation, and a user or account identifier.

Spans represent individual operations within a trace. A retrieval step is a span. A model inference call is a span. A tool invocation is a span. Each span carries its own start and end timestamps, status, and operation-specific attributes. Spans are nested to reflect parent-child relationships, allowing you to see that a model inference span spawned three tool call spans, one of which spawned its own model inference span.

The OpenTelemetry GenAI semantic conventions define specific span types and attributes to ensure AI telemetry is consistent and interoperable across providers and frameworks. Key attributes include gen_ai.operation.name for the type of operation, gen_ai.request.model for the model being called, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens for token consumption, and gen_ai.provider.name for the model provider. Datadog, Splunk, Grafana, and other major observability platforms have begun supporting these conventions natively, allowing teams to instrument once and analyze across platforms.

Events capture discrete occurrences within spans. The OpenTelemetry GenAI conventions define events for prompt inputs, model outputs, tool calls, and evaluation results. These events can carry structured payloads, but the conventions explicitly recommend that instrumentation not capture prompt and response content by default. Instead, content capture should be opt-in, with three recommended patterns. The first is not to record content at all, relying solely on metadata. The second is to record content directly on spans using attributes, which is suitable for pre-production environments where privacy constraints are relaxed. The third is to store content externally and record only references on spans, recommended for production environments where data volume and sensitivity require separate handling.

A well-structured telemetry schema for an AI interaction might look like this conceptually. The trace carries a trace ID, session ID, user ID, and environment. The root span represents the overall request, with attributes for the use case, request source, and prompt template version. Child spans represent retrieval with attributes for the number of documents returned and relevance scores. Another child span represents model inference with attributes for the model name, token counts, latency, and finish reason. Additional child spans represent tool calls, including their names, inputs, outputs, and durations. An evaluation event attached to the model inference span includes a score, the evaluator’s name, and the evaluation criteria.

This structure makes AI interactions searchable by any dimension. You can query for all requests that used a specific prompt version. You can filter for interactions where retrieval returned fewer than three documents. You can aggregate token usage by model and use case. You can correlate quality scores with specific prompt versions to measure the impact of changes.

Prompt versioning deserves special emphasis. Every system prompt, user prompt template, and few-shot example set should be version-controlled and referenced in telemetry by a version identifier. When a quality regression occurs, correlating it with a specific prompt version change is often the fastest path to the root cause. Without versioning, teams resort to comparing timestamps against deployment logs, a brittle and error-prone process.

Session and conversation management are equally important for conversational AI systems. A session ID groups related turns in a conversation, allowing you to analyze multi-turn interaction patterns, measure conversation-level success rates, and identify points of abandonment. Without session-level correlation, you can measure individual response quality but cannot understand the user journey.

Feedback and Evaluation Signals

Observability without quality measurement is like monitoring a factory’s power consumption without inspecting the products coming off the line. You know the system is running, but you do not know if it is producing good results.

Feedback and evaluation signals close this gap by providing continuous measurement of AI output quality. These signals come from three sources, each with distinct strengths and limitations.

User feedback is the most direct signal of real-world quality, but it is also the sparsest. Explicit feedback mechanisms such as thumbs-up and thumbs-down buttons, star ratings, or correction interfaces capture user satisfaction, but typically only a small percentage of users provide it. Implicit feedback, including acceptance or rejection of suggestions, editing of generated content, escalation to human agents, and session abandonment, provides higher coverage but requires careful interpretation. A user who edits an AI-generated email may be making minor stylistic adjustments or may be correcting fundamental errors. The edit distance or type of change can help distinguish these cases, but ambiguity remains.

Structure user feedback as events attached to the relevant model inference span in your telemetry. Include the feedback type, value, timestamp, and any associated user action. This allows you to correlate feedback with specific model outputs, prompt versions, and retrieved context, enabling analysis of what drives quality from the user’s perspective.

Automated evaluators provide the scale and consistency that human feedback cannot. These evaluators run programmatically against model outputs and produce structured scores on dimensions such as groundedness (does the output stick to provided context), relevance (does the output address the user's question), faithfulness (is the output factually consistent with source material), toxicity (does the output contain harmful content), and format compliance (does the output follow specified structural requirements).

Automated evaluators fall into two categories. Rule-based evaluators use pattern matching, heuristics, or deterministic checks to assess outputs. They are fast and consistent, but limited to well-defined criteria. LLM-based evaluators use a separate model to assess the primary model’s output, typically using a detailed rubric and chain-of-thought prompting to improve reliability. LLM-based evaluators can assess nuanced dimensions such as helpfulness and accuracy, but they also introduce their own costs and latency. A 2025 McKinsey survey found that 51% of organizations using AI experienced at least one negative consequence, underscoring the importance of systematic evaluation rather than reactive incident response.

The OpenTelemetry GenAI conventions include an evaluation event type that captures the evaluator name, score, score label, and the trace or response being evaluated. This standardization allows evaluation results from different sources and evaluators to be aggregated, compared, and analyzed within the same observability infrastructure.

Human review and annotation provide the highest-fidelity quality signal, but at the highest cost. Structured review workflows where subject matter experts assess AI outputs for accuracy, appropriateness, and domain correctness generate labeled data that can be used to calibrate automated evaluators, identify failure patterns, and build regression test sets. The practical approach is to route a risk-weighted sample of production outputs for human review, prioritizing high-stakes use cases, low-confidence predictions, and cases flagged by automated evaluators.

The combination of these three feedback sources creates a quality measurement system that operates at multiple levels of fidelity and coverage. Automated evaluators run on every request, providing baseline quality monitoring. User feedback supplements this with a real-world signal. Human review provides periodic deep assessment and calibration. Together, they form a feedback loop that connects observability to continuous improvement.

Sensitive Data Handling

Here is the challenge that makes AI observability fundamentally different from traditional application logging from a privacy perspective. In a conventional application, the sensitive data lives in database fields with well-defined schemas. You know which fields contain personal information because you designed the schema. In AI systems, sensitive data can appear anywhere. Users paste personal information into prompts. Retrieval systems surface documents containing confidential data. Models may generate outputs that include or infer sensitive information. The unstructured, free-text nature of AI interactions means that every prompt and every response is a potential vector for sensitive data leakage into your telemetry pipeline.

This is not a hypothetical concern. Research on telemetry pipelines has found that observability systems can inadvertently become repositories for personally identifiable information when they ingest the unstructured, information-rich prompts that users submit to AI systems. For organizations subject to GDPR, HIPAA, CCPA, or other data protection regulations, sensitive data in observability logs creates compliance exposure that extends well beyond the AI system itself.

Addressing this requires a defense-in-depth approach that applies controls at multiple layers.

Redaction before persistence

The most effective control is to detect and remove sensitive data before it reaches your logging backend. This should happen in the telemetry pipeline itself, not as a post-hoc cleanup process. Tools like Microsoft Presidio combine regular expression patterns with named entity recognition models to identify and redact names, email addresses, phone numbers, credit card numbers, social security numbers, and other PII categories. The OpenTelemetry Collector supports custom processors that can intercept traces and redact sensitive attributes before forwarding them to the observability backend. This approach, sometimes called “safe observability,” treats redaction as a first-class pipeline operation rather than an afterthought.

Field-level allowlists

Rather than trying to identify and remove all sensitive data from logged content, a more robust approach is to default to logging only structured metadata fields and explicitly allow specific content fields when needed. This inverts the typical logging model. Instead of logging everything and hoping your redaction catches sensitive data, you log only known-safe fields by default. A typical allowlisted event might include the trace ID, session ID, model name, prompt version, token counts, latency, finish reason, evaluation scores, and error codes, but not the raw prompt text, output text, or retrieved documents. This metadata alone is sufficient for most operational analysis.

Tiered access and break-glass procedures

Some debugging scenarios genuinely require access to full prompt and response content. Rather than logging this content for all requests, implement a break-glass mechanism that allows authorized personnel to access higher-fidelity logs under controlled conditions. This might involve a separate, access-controlled logging tier that captures the full content of a configurable sample of requests, with access gated behind approval workflows and audited via immutable access logs. The key is that full-content access is an exception requiring justification, not the default.

Pseudonymous identifiers

Replace user-identifying information with pseudonymous IDs in your telemetry. A hashed or tokenized user identifier allows you to correlate requests within a session and across sessions for the same user without exposing the user’s identity in the logging system. If you need to resolve a pseudonymous ID back to a real user, that resolution should require a separate lookup through an access-controlled service, creating an additional barrier against casual exposure.

Encryption and access control

Telemetry data at rest should be encrypted, and access should be restricted to roles that require it. This is standard practice for any sensitive data store, but it is worth emphasizing because observability platforms are often treated with less security rigor than production databases. If your AI logs contain prompts and responses, they deserve the same protection as your customer data, because they may very well contain customer data.

Retention policies

Define and enforce retention periods for different categories of telemetry data. Structured metadata might be retained for months or years to support trend analysis. Sampled full-content logs might be retained for days or weeks to support debugging. Evaluation data might be retained for the life of the model to support longitudinal quality analysis. Automated expiration reduces compliance risk by ensuring that sensitive data does not accumulate indefinitely. Critically, retention policies must apply to backups and replicas as well as primary storage. A deletion policy that applies to the primary store but not to the data warehouse or backup system provides only the illusion of compliance.

Third-party logging boundaries

Many AI systems integrate with external providers for model inference, retrieval, or tool execution. Understand what data is transmitted to these third parties and what they log on their end. Your redaction pipeline should sanitize data before it leaves your infrastructure, not after. This is particularly important for prompts sent to external model APIs, which may contain user data that you are contractually or legally prohibited from sharing with third parties.

The goal is not to eliminate logging. The goal is useful observability with data minimization enforced before data is stored. Security and compliance reviews will focus less on your dashboard and more on whether you can demonstrate that minimization, access control, retention, and deletion actually hold throughout the logging path.

Operational Use Cases

An observability stack is an investment. Like any investment, it must deliver returns that justify its cost. The following operational use cases demonstrate how AI observability translates into tangible business value.

Debugging and root cause analysis

When a user reports that the AI system gave the wrong answer, debugging without observability involves guesswork. With a properly instrumented stack, the response team can pull the trace for the specific interaction, inspect the prompt version and system instructions, examine the retrieved context and relevance scores, review the model’s raw output, and identify where the chain broke down. Was it a retrieval failure that surfaced irrelevant documents? A prompt issue that gave the model ambiguous instructions? A model limitation on a specific type of reasoning? Structured traces with semantic context transform debugging from an art into a systematic process.

Incident response

When AI quality degrades across a user population, observability enables rapid detection and triage. Monitoring dashboards built on evaluation scores, error rates, and user feedback can trigger alerts when quality metrics drop below defined thresholds. The ability to slice metrics by prompt version, model version, and time window allows teams to identify whether the degradation correlates with a specific change quickly. Organizations that have invested in AI observability report significantly faster mean time to resolution for AI-related incidents because they can isolate the contributing factor without resorting to trial-and-error rollbacks.

Prompt tuning and optimization

Prompt engineering is iterative work that benefits enormously from data. Observability enables teams to compare quality scores across prompt versions using production data rather than synthetic benchmarks. When you see that prompt version 3.2 produces higher groundedness scores but lower user satisfaction than version 3.1, you have actionable information to improve. Without this data, prompt optimization is driven by intuition and anecdote.

Evaluation and regression testing

Production observability data feeds directly into evaluation workflows. Sampled production traces become the test cases for regression suites. When a team proposes a prompt change or model upgrade, they can run the new configuration against real production scenarios and compare quality scores with the current baseline. This is how AI teams build the same kind of continuous integration and testing discipline that software engineering teams have relied on for decades, adapted for non-deterministic systems.

Compliance and audit readiness

Regulations such as the EU AI Act require organizations to maintain documentation on how their AI systems make decisions, what data they use, and how they are monitored. A well-structured observability stack provides the evidentiary foundation for compliance. Traces demonstrate the decision-making process. Evaluation logs demonstrate ongoing quality monitoring. Access logs demonstrate that sensitive data is appropriately protected. Organizations that build observability with compliance in mind from the start will find audit preparation far less burdensome than those that attempt to reconstruct this evidence after the fact.

Cost management

Token usage telemetry, aggregated by use case, model, and prompt version, provides the visibility needed to manage AI infrastructure costs. Teams can identify use cases with disproportionate token consumption, detect prompt patterns that unnecessarily inflate context windows, compare cost-per-quality-point across models, and set budget alerts that trigger when usage trends exceed forecasts. As AI workloads scale, cost management becomes a strategic concern, and observability is the only reliable mechanism for maintaining visibility into how the money is spent.

Building the Foundation

The observability stack for AI is not a single tool or platform. It is an architectural layer that integrates telemetry capture, structured storage, evaluation pipelines, privacy controls, and operational workflows into a coherent system. The organizations doing this well share several characteristics.

They treat observability as a first-class requirement, not an afterthought that gets bolted on after deployment. They design their telemetry schema before writing application code, ensuring that the instrumentation supports the analysis they will need. They adopt open standards, particularly OpenTelemetry’s GenAI semantic conventions, to avoid vendor lock-in and enable interoperability. They enforce privacy controls in the pipeline, not in policy documents. And they connect observability to action, using evaluation signals to drive prompt improvements, model selection decisions, and incident response.

The tooling landscape is maturing rapidly. Platforms such as Arize, Langfuse, Datadog LLM Observability, Splunk, and others now offer purpose-built capabilities for tracing AI interactions, evaluating output quality, and monitoring production behavior. The OpenTelemetry community’s work on GenAI semantic conventions is establishing an interoperability layer that allows organizations to instrument once and analyze across multiple backends.

But tools alone are not enough. The observability stack requires organizational commitment to instrumentation discipline, evaluation rigor, and privacy hygiene. It requires that engineering teams adopt new practices for prompt versioning and telemetry schema design. It requires governance teams to define retention policies and access controls that balance debugging utility and privacy protection. And it requires that leadership recognize observability as a strategic capability that protects the organization’s AI investments.

The organizations that build this foundation now will operate with a decisive advantage. They will debug faster, improve quality systematically, manage costs proactively, and demonstrate compliance with confidence. Those who defer will find themselves operating AI systems they cannot explain, audit, or improve with any rigor.

You would not operate a production database without monitoring. You would not deploy a web application without error tracking. The bar for AI systems should not be lowered. If anything, given the non-deterministic nature of these systems and the sensitivity of the data they process, the bar should be higher.

Start with structured metadata on every request. Add evaluation signals. Enforce redaction before persistence. Build from there.

The observability stack for AI is not optional infrastructure. It is the difference between deploying AI and operating it responsibly.