LLM Observability Explained: Monitoring AI Apps in Production
LLM observability is the practice of measuring how a generative AI application behaves in real use: which models and tools it calls, what those operations cost, how quickly they complete, whether outputs satisfy requirements, and where failures originate. It combines traces, metrics, evaluations, feedback, and operational safeguards so teams can improve an AI system using evidence rather than guesswork.
A chatbot can return a technically successful API response and still give an unsupported answer. A research assistant may retrieve the wrong document; a coding agent may repeatedly call a failing tool; a customer-support bot may quietly become more expensive after a model change. LLM observability makes those application-level problems easier to detect and investigate. It does not, by itself, prove that every answer is correct or reveal an AI model's private internal reasoning.
This guide explains what to monitor, how traces and evaluations work, which tools are available, and how to establish a practical monitoring process without recording sensitive user information unnecessarily.
LLM Observability in 60 Seconds
| Question | Practical answer |
|---|---|
| What is monitored? | Model and tool calls, request latency, errors, token usage, cost, retrieval results, output quality, and safety events |
| What is a trace? | A linked record of instrumented operations that contributed to an application request |
| What is an evaluation? | A test or judgment against a defined standard, such as correct JSON, accurate citations, or task completion |
| What is different from ordinary monitoring? | An HTTP success can still contain an incorrect or unhelpful AI response |
| Do traces reveal hidden chain-of-thought? | No. They reveal only instrumented, available data; private model reasoning is not generally exposed |
| Must you log full prompts? | No. Metadata-only tracing, masking, controlled capture, and limited retention are often preferable |
| What is a useful starting point? | Track errors, model/version, duration, token cost, a minimal trace, and a small human-reviewed evaluation set |
A useful mental model: Observe the workflow → detect an issue → inspect its cause → test a correction → measure whether the correction worked.
What LLM Observability Actually Means
A large language model (LLM) generates a response using the inputs supplied to it. The surrounding application may retrieve documents, construct a prompt, choose a model, call tools, validate output, and display the result. Observability concerns this whole observable system, not just a model endpoint.
For example, a support assistant might answer a refund question by searching company policies, assembling relevant excerpts, calling a language model, and converting the answer to a structured response. The model might behave as instructed but still receive a policy that has been superseded. Without tracing and evaluation, the failure may appear to be a mysterious model mistake.
Observability Is Not a Single Dashboard
A robust observability practice connects four kinds of evidence:
- Metrics describe patterns: latency percentiles, error rates, token totals, and average cost.
- Traces reconstruct instrumented steps for individual requests.
- Evaluations estimate whether the outputs and intermediate steps meet defined quality criteria.
- Feedback and incident review connect technical signals to what users experienced.
These signals answer different questions. An application can have excellent uptime, low token costs, and unacceptable answer quality simultaneously.
Observability Is Not Full Access to Model Reasoning
A trace may show a model request, returned text, tool name, retrieved document IDs, retry count, or explicit decisions written by application code. It does not automatically expose a model's private hidden reasoning or give a complete causal explanation of why the model chose every token. Even when an application records an agent's visible plan, that plan is not necessarily a faithful account of every internal computation.
The more accurate promise is visibility into instrumented execution and measurable outcomes, which is already enough to resolve many production problems.
Why Ordinary Software Monitoring Is Not Sufficient
Traditional application performance monitoring (APM) remains useful. It tracks infrastructure availability, network failures, database performance, exceptions, and endpoint latency. AI applications need those signals too; the issue is that standard operational success does not establish semantic correctness.
Consider a server that returns HTTP 200 in 900 milliseconds. From an infrastructure perspective, it worked. But its AI answer might cite a nonexistent policy, omit the customer's actual question, or recommend an unavailable product. None of those mistakes necessarily produces an exception.
| Dimension | Conventional APM | LLM-specific observability |
|---|---|---|
| Reliability | Errors, timeouts, saturation | Model failures, malformed outputs, tool and agent failures |
| Performance | Server duration and throughput | Time to first output, model call duration, retrieval time, end-to-end response |
| Spending | Infrastructure and network usage | Input/output tokens, cache usage, model and evaluation cost |
| Quality | Business checks where available | Relevance, grounding, task success, policy adherence, human ratings |
| Debugging | Exceptions and service traces | Prompt assembly, retrieved evidence, model/version, tool spans, evaluation results—when instrumented |
This is not a claim that conventional software is always deterministic or that AI never returns ordinary errors. Both kinds of software can exhibit unpredictable failures. LLM observability extends rather than replaces APM.
The Four Signals to Monitor First
1. Reliability and Correctness of the Workflow
Begin with facts that can be checked automatically. Record model API errors, provider timeouts, malformed structured responses, missing required fields, unsuccessful tool calls, exceeded budgets, retry counts, and whether a task finished. A generated JSON object that violates a required schema is a clear error even if the text reads naturally.
Define the unit of success carefully. In a multi-step agent, an individual model call may succeed while the overall task fails because the final file was not created or an authorized action was never completed.
2. Latency and User Experience
Measure end-to-end latency from the user's request to the completed useful response, and break it down into stages. For streaming systems, time to first token (TTFT) and output generation speed can help explain the experience; for tool-heavy agents, retrieval, external API calls, and retries may matter more.
Track median (p50), p95, and where justified p99. A 1.5-second first response may feel slow for voice conversation but acceptable for some research tasks. No single TTFT threshold fits every product, device, model, and workflow.
3. Usage and Total Cost
During AI inference, usage depends on the provider and model. Track billable input and output tokens where supplied, cached-token categories when available, tool charges, retries, evaluation expenses, and cost per completed useful task.
A team may reduce cost per model call while increasing total spending if retries or multiple agent steps multiply. Reporting only the cheapest-looking API price can conceal that failure.
4. Output Quality and Safety
Quality needs task-specific definitions. A document assistant might require sourced answers and correct page references; an extraction workflow needs field accuracy; a coding agent needs passing tests; a support chatbot should answer the customer without breaking company policy.
An AI system can generate convincing hallucinations without producing an error message. Detecting such problems requires a combination of deterministic rules, evaluation samples, references, and sometimes human review. It cannot be inferred from latency or token use alone.
How LLM Tracing Works: Requests, Spans, and Sessions
A trace follows one end-to-end operation through its instrumented components. Each individual operation is commonly represented by a span: a timed event with a name, status, attributes, and a relationship to its parent span. Several requests may also be associated with a longer conversation or user session when appropriate identifiers are available.
Imagine the question, "What is our refund policy for international orders?" The application might produce this execution tree:
Trace: customer-refund-question
1. Classify request 18 ms
2. Search knowledge base 142 ms
3. Assemble context 11 ms
4. Call LLM 1,460 ms
5. Validate answer 24 ms
6. Return response
The durations above are invented to illustrate the trace structure, not measurements of a particular product. A useful trace may include model version, operation result, retrieved document identifiers, token counts, retry events, and error codes. Access to actual prompt bodies or documents should be controlled separately.
Find the Failure, Not Just the Final Symptom
If an answer cited an outdated returns policy, inspect whether the retriever selected the wrong version, the prompt omitted its publication date, or the model disregarded the supplied text. Tracing narrows the investigation. It does not always prove which component was responsible, particularly when evidence is missing.
Use Common Telemetry Conventions Where Possible
The OpenTelemetry Semantic Conventions for Generative AI specify names and structures for many GenAI-related spans and metrics. They include model operations, token usage, retrieval, and tool-related activity. The conventions continue to evolve, so instrumentation versions and supported attributes should be checked against current documentation.
A benefit is interoperability: telemetry can be exported using supported OpenTelemetry pipelines rather than relying entirely on a proprietary data format. But a common format does not make different tools identical in evaluation, storage, or privacy behavior.
What Should You Capture in a Production Trace?
Start with low-risk metadata. Add full content only if it is justified, permitted, and properly protected.
| Field | Why it helps | Suggested handling |
|---|---|---|
| Request / trace ID | Correlates stages of an incident | Store by default; avoid exposing it publicly |
| Model and version | Detects regressions after model changes | Store when available |
| Call duration and status | Shows slow and failed operations | Store by default |
| Token counts and estimated cost | Explains usage and billing | Store when available; label estimates |
| Tool names and status | Helps identify failing integrations | Record minimal operational metadata |
| Retrieved document IDs / versions | Supports source and freshness debugging | Prefer identifiers to full confidential content |
| Prompt, output, or tool arguments | Can help reproduce complex failures | Opt in selectively; redact, restrict, and set retention limits |
| User feedback | Links traces to perceived quality | Collect proportionately and interpret cautiously |
The official OpenTelemetry GenAI specification advises against recording sensitive prompt and output content by default. This is a critical correction to the common assumption that a useful observability platform must copy every conversation into a third-party dashboard.
Evaluation: Monitoring Whether Answers Are Good
A trace answers what ran. An evaluation asks whether the outcome met a defined requirement. Good engineering teams need both.
Offline Evaluations Before Deployment
Build a dataset of representative tasks with known expectations. For example, a refund assistant's test set might contain routine returns, international orders, damaged items, expired return windows, and ambiguous requests. Score new model or prompt versions against the same cases before release.
This is especially important when changing prompts, retrieval logic, RAG configuration, model providers, or routing policies. A small change may improve easy cases while quietly making edge cases worse.
Online Evaluations After Deployment
Production monitoring samples or selectively evaluates real interactions under an approved data policy. It can reveal new questions and failure patterns that an offline dataset missed. For example, a rise in unanswered shipping questions after a policy update may justify adding fresh cases to regression tests.
Online scoring should not automatically trigger a production rollback just because one evaluator score drops. Review sample size, evaluator reliability, business consequences, and whether the observed shift is statistically or operationally meaningful.
Use Deterministic Checks Whenever They Can Answer the Question
A schema validator can determine whether a response is valid JSON. A calculator can check arithmetic. A database lookup can confirm that an order ID exists. A test runner can determine whether submitted code passes defined tests. These checks are generally more directly verifiable than asking another generative model whether a requirement was met.
Use an evaluator model for more subjective or semantic criteria only when simpler checks are inadequate.
LLM-as-a-Judge: Powerful, but Not an Independent Source of Truth
LLM-as-a-Judge uses a model to assess another model's response against a rubric or reference material. For example, a judge can examine a customer question, the retrieved refund policy, and the chatbot's answer, then assess whether the answer is grounded and relevant.
Write a Testable Rubric
Avoid vague instructions such as "Is this a good answer?" Instead, define criteria:
- Did the answer address the user's actual question?
- Are dates, monetary amounts, and policy conditions supported by the supplied reference?
- Does it avoid inventing exceptions?
- Did it correctly acknowledge uncertainty or missing data?
- Does it follow the specified response structure?
Keep relevance, grounding, format, and safety as separate measurements when diagnosis matters. One combined score can hide a serious factual error behind otherwise strong writing.
Calibrate Judges Against Human-Reviewed Examples
Evaluator models can exhibit position bias, verbosity preferences, inconsistent grading, and difficulty detecting errors they would make themselves. Research on MT-Bench and LLM-as-a-Judge describes important limitations, while research on judging without human grounding shows why reference answers can matter.
Create a small labeled set reviewed by knowledgeable people; compare the judge's decisions against it, examine false positives and false negatives, and retest when the judge model or rubric changes. An evaluator score is a fallible quality signal, not proof of factual correctness.
How Much Traffic Should Be Evaluated?
There is no universal 5% or 10% rule. Evaluation coverage should follow risk, traffic, cost, and the size of the quality shift the team wants to detect. A low-risk writing helper may benefit from sampled evaluation. A sensitive workflow may require deterministic checks on every transaction, plus selective human review. Evaluate cost and risk together.
A second evaluation call also does not necessarily double total inference spending; the actual added expense depends on evaluator model, prompt length, sample rate, and whether the result is computed asynchronously.
RAG Observability: Look Before and After Retrieval
A retrieval-augmented generation (RAG) application uses information retrieved from an external collection to support its answers. Its failures may come from retrieval, context assembly, source freshness, or generation. Observability should distinguish these stages.
The RAG Quality Triad
- Context relevance: Did retrieval return passages that address the question?
- Groundedness or faithfulness: Are claims in the answer supported by the retrieved material?
- Answer relevance: Does the final response answer the question asked?
These criteria are useful, but a grounded answer may still be wrong if the underlying documents are wrong or out of date. Measure source freshness and authority separately when they are essential.
Retrieval Measurements That Help Diagnose Problems
For an evaluated sample with labeled relevant documents, metrics such as recall@k or precision@k can help assess retrieval. Where labels are unavailable, inspect retrieved document IDs, ranking, query rewrites, version dates, and human-reviewed cases. If your system depends on a vector database and embeddings, record which embedding or indexing version was used so that a change can be connected to a quality regression.
Worked example: A customer asks about a 30-day return window. The retriever chooses a policy from two years ago stating 14 days. The model accurately summarizes that outdated passage. Answer relevance and local grounding may appear good, yet the final business answer is wrong. The fix is document lifecycle management and retrieval filtering—not merely a more capable model.
AI Agent Observability: Monitor Tools and Consequences
An AI agent may plan steps, call APIs, inspect results, and retry. As workflows grow more capable, their execution becomes less like a single chatbot completion and more like a distributed application. For agents that interact with browser interfaces, see Mozzim's guide to computer-using AI agents.
Trace Every Consequential Tool Operation
Useful fields include tool name, timing, status, sanitized arguments, retries, result identifiers, and whether authorization was checked. This is especially important for actions such as sending an email, modifying records, issuing refunds, or purchasing products. An agent's claim that an action succeeded should not substitute for confirmation from the external system.
Prevent Runaway Loops
Set explicit maximum model calls, tool calls, execution duration, and monetary budgets where supported. Monitor repeated identical calls, failed retries, growing token consumption, and tasks that never reach a completion condition.
Observability can detect warning signs, but stopping harmful execution requires enforced controls in the application or infrastructure. A passive dashboard alone cannot halt an agent.
Measure Task Completion, Not Just Plausible Dialogue
For a report-writing agent, success may mean producing the requested file with all sections. For a booking assistant, it may mean receiving a confirmed reservation rather than simply opening a checkout page. For customer support, it may mean resolving the ticket without violating policy. Tool execution success and user-goal success are separate metrics.
Worked Production Incident: An Assistant Gives an Outdated Answer
Imagine a hypothetical internal help-desk assistant that has been returning correct expense-policy answers for months. After new policy documents are uploaded, users begin reporting incorrect meal-reimbursement limits.
Step 1: Detect the Pattern
A sampled groundedness evaluator flags more answers that conflict with a current reference. Support staff also report an increase in corrections. Treat those signals as an investigation trigger, not immediate proof that the model has deteriorated.
Step 2: Inspect a Representative Trace
The trace shows a successful document-search operation, a relevant-looking retrieved passage, and a generated answer. But the passage's document version is two years old. The model accurately reproduced an obsolete rule.
Step 3: Identify a Testable Cause
Investigation finds that the ingestion pipeline kept superseded policy documents active without a date filter. The issue is an index-management failure, not necessarily a new hallucination pattern inside the model.
Step 4: Correct and Test
The team marks superseded documents inactive, adds policy-effective-date constraints, and tests a curated set of old and new reimbursement scenarios. It verifies that the current policy is selected and that the model cites the right passage.
Step 5: Confirm Improvement in Production
Continue monitoring retrieval-version errors, sampled factual correctness, support corrections, and latency. The process is complete only when the change improves real outcomes without introducing a new problem.
Privacy and Security: Do Not Turn Observability Into a Data Leak
Tracing can create a second copy of the most sensitive information in your application: private conversations, customer records, authentication secrets, internal documents, or tool outputs. Logging all of it to a third-party service simply because debugging is easier can create significant privacy and security exposure.
Apply Data Minimization First
Choose whether a field needs to be stored at all. For many performance and cost questions, model name, request ID, status, timing, and token counts are enough. For debugging retrieval, a document ID and version may be sufficient without copying the whole document.
Treat Redaction as Defense in Depth
Mask or filter sensitive data before export when possible, but recognize that regex rules can miss unusual formats or contextual secrets. Access controls, encryption, retention limits, deletion workflows, audit logs, and appropriate deployment locations remain important. Review third-party processor agreements and applicable requirements for sensitive data.
Tools such as Langfuse document controls for masking trace fields; the correct setting depends on the SDK and infrastructure in use.
Make Human Access Proportionate
Engineers who need aggregated latency should not necessarily have access to customers' raw messages. Restrict sensitive trace access by role, make investigation access auditable, and use anonymized or synthetic test cases whenever they can reproduce the issue.
Which LLM Observability Tools Should You Consider?
No single tool is best for every team. Many conventional APM products now support GenAI-specific telemetry as well, so evaluate functionality rather than treating 'APM' and 'LLM observability' as mutually exclusive categories.
| Tool / approach | Main relevance | What to verify before adopting |
|---|---|---|
| OpenTelemetry | Open conventions and export pipelines for traces and metrics | GenAI semantic-convention version, instrumentation coverage, backend support |
| Langfuse | LLM traces, costs, evaluation and prompt workflows | Hosted versus self-hosted needs, masking, usage-based limits |
| Arize Phoenix | Traces, datasets, experimentation and evaluators | Deployment model, integrations, evaluator requirements |
| LangSmith | Tracing, offline/online evaluation and regression tests | SDK compatibility, privacy settings, evaluator costs |
| Existing APM platform | Correlating AI operations with service infrastructure | Depth of GenAI span support, quality evaluation, retention |
Explore the LangSmith evaluation guide, Arize Phoenix overview, and Langfuse cost-tracking documentation for product-specific detail. Features, hosting options, and prices change, so review current vendor documentation before making a purchasing decision.
How to Implement LLM Observability in Seven Steps
Step 1: Define Your Most Important Failure
Begin with a specific business or user risk: unsupported answers, expensive agent loops, broken JSON, slow responses, or failed customer actions. Decide what a correct outcome looks like. Observability works best when teams know which failure they need to distinguish.
Step 2: Instrument the End-to-End Path
Assign trace IDs and instrument retrieval, prompt assembly, model calls, tools, validation, and the final result. Start with minimal metadata. Verify that parent-child relationships are preserved so an incident can be followed through the system.
Step 3: Establish a Small Evaluation Dataset
Collect consent-appropriate representative examples, including challenging edge cases. Document reference outputs, acceptance criteria, and which judgments require human expertise. Avoid relying only on demonstrations where the system already succeeds.
Step 4: Add Layered Checks
Use deterministic checks for schema, required fields, budgets, and permissions. Add model-based judges for semantic criteria only where justified, then compare their classifications against human-reviewed examples. Track both errors the evaluator catches and failures it misses.
Step 5: Build Focused Dashboards
Display p50/p95 latency, error rate, token cost, cost per successful task, retrieval failures, and the quality metrics that matter to the product. Segment by model version, feature, and release when useful. Avoid collecting personally identifiable labels solely for easier dashboard filtering.
Step 6: Define Actionable Alerts and Hard Limits
Page engineers for incidents that need immediate intervention, such as runaway tool loops, high failure rates, exceeded spending budgets, or failed authorization checks. Route low-confidence judge findings to review queues or periodic reports unless their impact justifies stronger response.
Step 7: Close the Loop With Regression Testing
When a real failure is confirmed, add a safe, reproducible version of that case to your evaluation dataset. Test the fix before deploying it and compare production results after release. This turns monitoring into continuous improvement rather than an archive of incidents.
A Simple Cost Example: Monitoring the Monitoring System
Suppose an application processes 200,000 requests a month, averaging 2,500 input tokens and 500 output tokens per request. For illustration only, assume the primary model costs $1 per million input tokens and $4 per million output tokens.
- Input tokens: 200,000 × 2,500 = 500 million → $500.
- Output tokens: 200,000 × 500 = 100 million → $400.
- Illustrative primary-model cost: $900 per month, excluding other charges.
Now suppose the team evaluates 5% of requests (10,000 cases) using an evaluator that receives 2,000 input tokens and produces 250 output tokens. At illustrative evaluator prices of $2 per million input tokens and $8 per million output tokens, the judge cost is $60. Additional storage, telemetry processing, human review, and other operations are extra.
The lesson is not that 5% is the ideal sample rate or that every deployment costs this amount. It is that observability cost should be measured as part of total cost, with explicit assumptions and room for more intensive checking of higher-risk tasks.
Common Mistakes That Make Observability Less Useful
Logging Everything Without a Privacy Plan
Raw prompt capture can make debugging easier and expose sensitive information. Decide what to retain before enabling content-rich traces; do not assume storage is justified because the data is technically accessible.
Treating a Judge Score as Factual Proof
A positive judge score may reflect the evaluator's own blind spots. Require independent checks for consequential information and periodically recalibrate with human-reviewed cases.
Measuring Only Token Cost
A cheap failed response that needs three retries may cost more per successful task than a stronger model that gets it right once. Include user outcomes and failure recovery.
Ignoring Retrieval and Tool Versions
Models are often blamed for errors caused by outdated documents, stale APIs, or incorrect tool arguments. Record which components and versions actually participated in the request.
Using Alerts That Nobody Can Act On
Hundreds of noisy warnings train people to ignore all warnings. Connect alert severity to user harm, a defined owner, and a documented response procedure.
Assuming Tracing Automatically Stops Harm
A trace can document a costly retry loop, but preventing it requires enforceable application-level call limits, timeout rules, budgets, and authorization checks.
Frequently Asked Questions
Is LLM observability the same as model evaluation?
No. Observability combines execution data, metrics, traces, evaluations, and feedback. Evaluation is the part that measures whether behavior meets specific quality criteria.
Can traditional monitoring tools track LLM applications?
Yes. Standard monitoring remains important, and some APM platforms support GenAI-specific spans and metrics. The practical requirement is to capture the relevant AI operations and quality signals, regardless of the tool's category.
Can tracing show an AI model's private chain-of-thought?
Not inherently. Traces reveal operations and data that application code and providers expose and that instrumentation records. They should not be described as unrestricted access to internal model reasoning.
Should an application store full prompts and responses?
Only when necessary and permitted under an appropriate security and privacy policy. Operational metadata, redacted fields, and document identifiers are often sufficient; sensitive content capture should be an intentional choice.
Can an LLM judge reliably detect hallucinations?
It can help flag unsupported claims against supplied references, but it can miss errors, misread evidence, and inherit biases. Combine it with curated test cases, deterministic verification where possible, and human review.
Is observability expensive for a small AI product?
It can be lightweight at first. Start with instrumented timing, token usage, error events, minimal traces, and manual reviews. Add paid tooling or automatic judging when its value exceeds its cost and operational burden.
What should I monitor for autonomous agents?
Tool calls and results, retries, elapsed time, spending, authorization checks, completion of the user's actual goal, and recovery from failures. Add enforced execution limits rather than relying on alerts alone.
Conclusion: Make AI Behavior Measurable and Correctable
LLM observability is not about collecting the maximum amount of AI-generated text or claiming to read a model's mind. It is about understanding what your application did, how much it cost, whether it completed the intended task, and where a mistake may have originated.
A good implementation starts small: instrument the critical workflow, choose meaningful success criteria, track reliability and cost, evaluate representative outcomes, protect private information, and turn confirmed failures into regression tests. Add more sophisticated judges, tracing detail, and agent monitoring only when the product's complexity and risk justify them.
If you're building or managing a production AI application, useful observability is the difference between hoping your system works and having evidence that helps you improve it.
Continue Learning on Mozzim
Explore how AI agents operate, then revisit why AI hallucinations happen and the fundamentals of AI inference. For document-based assistants, our RAG beginner's guide provides the conceptual foundation for understanding retrieval and grounded answers.
Authoritative Sources and Further Reading
- OpenTelemetry — Generative AI semantic conventions: terminology for traces and metrics, with guidance on sensitive content capture.
- OpenTelemetry — Inside the LLM Call (2026): tracing GenAI applications using OpenTelemetry.
- LangSmith — Evaluation types: offline, online, code-based and model-based evaluations.
- Arize Phoenix — Evaluation documentation: traces, datasets, and evaluator workflows.
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: research on LLM judge capabilities and limitations.
