Short answer: a small team needs one trace per user request, one log row per model call, a cost figure on every row, and three or four cheap quality signals. That is enough. Log the model, token counts, latency, status, tool calls, prompt version and a tenant ID. Add cost, cache hits and retries. Score a sample of answers with user feedback, format checks and a small test set. Alert on cost per request, error rate, retries and loop length. Keep prompt and response text out of the logs unless you have a reason, a retention limit and redaction. You can do all of this with open tooling in about a week.
This guide is for small teams that ship an AI feature and cannot yet answer "why did that cost so much?" or "is it getting worse?". It pairs with our guide to why LLM bills grow faster than usage and the AI spend audit. It is engineering guidance, not legal advice.
What is LLM observability, and why is normal monitoring not enough?
Normal monitoring tells you a request returned 200 in 900 ms. An LLM feature can do that and still be wrong or far dearer than yesterday. The same input can give a different answer, one user action can fan out into several model calls and tool calls, and cost depends on tokens, not requests.
LLM observability adds three things on top of ordinary logs and metrics:
- Traces that show every step behind one user request: retrieval, each model call, each tool call, and the final answer.
- Cost and usage per call, so you can see which feature, tenant or prompt version is driving spend.
- Quality signals, so you can tell "the answer was wrong" from "the system was slow".
What should you log for every LLM call?

Start with the fields below. Most of them are metadata, so they carry little privacy risk. Where the OpenTelemetry GenAI semantic conventions already name a field, we show the attribute. That convention set is an emerging standard: its status is Development, not stable, and the project has moved to its own repository (open-telemetry/semantic-conventions-genai). Pin the version you use and expect renames.
| Field | Why you need it | OpenTelemetry attribute (emerging) |
|---|---|---|
| Trace ID and parent span | Why you need it: Ties a model call to the user request and to the steps around it | OpenTelemetry attribute (emerging): Standard trace context |
| Operation type (chat, embeddings, tool) | Why you need it: Lets you separate generation cost from retrieval cost | OpenTelemetry attribute (emerging): `gen_ai.operation.name` |
| Provider and model | Why you need it: Needed to price the call and to spot a model change | OpenTelemetry attribute (emerging): `gen_ai.provider.name`, `gen_ai.request.model` |
| Input and output tokens | Why you need it: The base for every cost figure | OpenTelemetry attribute (emerging): `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` |
| Cached input tokens | Why you need it: Shows whether prompt caching is working | OpenTelemetry attribute (emerging): `gen_ai.usage.cache_read.input_tokens` |
| Latency and status | Why you need it: Finds slow steps and failures | OpenTelemetry attribute (emerging): Span duration and status |
| Prompt version and feature name | Why you need it: Links a cost or quality change to a release | OpenTelemetry attribute (emerging): Your own attribute |
| Tenant or customer ID, hashed or internal | Why you need it: Finds the heavy users and per-customer margin | OpenTelemetry attribute (emerging): Your own attribute |
| Retry count and loop step number | Why you need it: Shows hidden repeat spending | OpenTelemetry attribute (emerging): Your own attribute |
| Computed cost in rupees or dollars | Why you need it: Lets finance and engineering read the same number | OpenTelemetry attribute (emerging): Your own attribute |
Do not depend on the OpenTelemetry metric names for now, because they are changing. Keep your own names in a thin wrapper.
Should you store prompts and responses?
Only with a reason. Prompt and response text is where personal data, customer secrets and credentials end up. The OpenTelemetry conventions treat message content (`gen_ai.input.messages` and `gen_ai.output.messages`) as opt-in for this reason. Treat your own logging the same way.
- Default to metadata only. Tokens, timings, IDs and scores answer most cost and reliability questions.
- Capture text for a small sample, such as failed calls, thumbs-down calls and a random slice.
- Redact before you store. Strip names, phone numbers, emails, PAN and Aadhaar numbers, and API keys. See our guide to redacting PII before LLM calls. Redaction reduces leakage. Test its recall on your own data before you rely on it.
- Set a retention period and enforce it. A trace store is a data store. If it holds personal data, it needs the same access control, erasure and log retention thinking as the rest of your system.
- Know where it sits. A hosted tracing tool that stores personal data is usually acting as your Data Processor under the DPDP Act. From 13 May 2027 you, as the Data Fiduciary, stay responsible for that data and need a valid contract with the vendor (s.8(1) and s.8(2)). How this applies to your setup depends on the facts, so check it with counsel.
Our DPDP-aware AI architecture guide covers where to place redaction and logging in the request path.
Which tool should a small team use?
You have three sensible routes. None of them is wrong. Pick the one that fits your data rules and your appetite for running software.
| Route | Good for | Watch out for |
|---|---|---|
| Hosted LLM tracing tool | Good for: Fast start, a UI for traces and scores, no servers to run | Watch out for: Text may leave your network. Check regions and data terms |
| Self-hosted open-source tracing tool | Good for: Data stays on your infrastructure; no per-unit fee | Watch out for: You run and patch it, and you pay for the database and storage |
| OpenTelemetry into the stack you already have | Good for: One place for app and LLM telemetry; no new vendor | Watch out for: Conventions are still moving, and you build the cost and quality views yourself |
To give one concrete public price point: Langfuse Cloud lists a free Hobby plan (50k units a month, 30 days of data access), a Core plan at $29 a month and a Pro plan at $199 a month, with Enterprise at $2,499 a month. Core and Pro include 100k units and then charge $8 per 100k. Data access is 90 days on Core and 3 years on Pro. Its cloud regions are US, EU and Japan, with no India region. Self-hosting it is free. Those are public list prices checked on 20 November 2026 (https://langfuse.com/pricing) and they change. This is an example, not a recommendation.
If customers require data to stay in India, a hosted tool in another region may fail that requirement for prompt text. Self-host it, or send metadata only. If you run models on your own servers, keep the tracing there too. See Own Your AI.
How do you trace a multi-step request or an agent?
Create one trace per user request. Make each model call, retrieval and tool call a child span. Pass the trace ID through queues and background jobs.
For agents, add these to every span:
- Step number within the run, and the run's step limit.
- Tool name, arguments hash and result status. Store the arguments themselves only under the text rules above.
- Stop reason: finished, step limit hit, error, timeout, or handed to a human.
- Identity used by the tool call.
- Approval event, if a human confirmed or rejected the action.
Two limits belong in code, not in a dashboard: caps on steps and retries per request, and a spend ceiling per request and per tenant per day. The logs then show how often the caps fire.
What cost signals should you watch?
Cost per model call is easy. Cost per user outcome matters more: sum the cost of every span in a trace and divide by completed outcomes, such as resolved tickets.
Watch these, in this order:
- Cost per request, as a distribution. Look at the median and the 95th percentile. A few runaway requests hide inside an average.
- Input tokens per request by conversation turn. Rising input tokens mean history or retrieved context is growing.
- Output tokens. At the list prices we checked on 20 November 2026 for OpenAI and Anthropic, output costs five times input per token. Long answers and hidden reasoning tokens add up.
- Cache hit rate. Divide cached input tokens by total input tokens. Cached reads are billed at a fraction of the normal input price (our AI spend audit lists the current multipliers and a worked example). A low hit rate on a long, repeated prompt means something near the top of the prompt changes on every call.
- Calls per user action and retries, since loops and retries repeat full prompts.
- Cost per tenant. The customers whose usage costs more than they pay.
- Batch share. Work that is not live can often go through a batch API. Anthropic's and OpenAI's lists show Batch at 50% off.
Put unit prices in a small config table with an effective date, and compute cost in your wrapper. Prices change: Google's Gemini 4 Carbon list price is scheduled to rise from $2 to $4 per million input tokens on 1 December 2026. A dated table lets you re-cost old traces correctly.
What quality signals can a small team collect cheaply?

Layer these, cheapest first:
- Hard checks on every response. Does the output parse as the JSON you asked for? Did the model refuse? Did it hit the length limit? Did it cite a document that was not retrieved? These are free and catch real failures.
- User signals. Thumbs up or down, a regenerate click, a copy click, an edit before sending, a follow-up that repeats the question. Treat them as signals, not truth.
- Retrieval checks for RAG: was any document retrieved, how many, and did the answer use it? An answer with no supporting source is a risk flag.
- A golden set. A few dozen real, anonymised cases with known good answers, run on every prompt or model change. Add each production failure to it once you have fixed it.
- An LLM judge on a sample. A second model scores answers against a rubric. Check the judge against human labels on a few dozen cases before trusting it, and re-check when you change the judge.
- Human review of a small weekly sample, including the worst-scoring answers.
Store each score against the trace ID, prompt version and model. Then "quality fell" becomes answerable: after which release, for which tenant, on which questions.
Why does an AI feature work in the demo and fail in production?
Demos use clean inputs, short contexts, one user and a lucky run. Production has messy inputs, long histories, many tenants, provider outages and unexpected use. You can observe most of that gap, then close it.
| Gap | What you see in production | Signal to add | Fix to try |
|---|---|---|---|
| Inputs differ from the demo | What you see in production: Odd formats, other languages, very long text, empty fields | Signal to add: Input length and language distribution | Fix to try: Validate and normalise inputs; add real cases to the golden set |
| No timeout or retry design | What you see in production: Hung requests, duplicate actions | Signal to add: Latency percentiles, retry count, timeout rate | Fix to try: Timeouts, bounded retries with back-off, idempotency keys for actions |
| Context grows over a session | What you see in production: Slow, costly, then confused answers | Signal to add: Input tokens by turn | Fix to try: Summarise or trim history; cap context |
| Retrieval returns the wrong thing | What you see in production: Confident answers from the wrong document | Signal to add: Retrieved count, source IDs, empty-retrieval rate | Fix to try: Fix chunking and permissions; answer "I do not know" when nothing is found |
| Prompt or model changed silently | What you see in production: Behaviour shifts with no code release | Signal to add: Prompt version and model on every span | Fix to try: Pin model versions; run the golden set before changing |
| No fallback when the model is wrong | What you see in production: Bad answers reach the user or the database | Signal to add: Validation failure rate, human-override rate | Fix to try: Output validation, a human check for risky actions |
Our AI product development team does this kind of hardening work.
How do you evaluate and monitor an AI agent?
An agent makes decisions between the user and the result, so monitor decisions and not only answers. Do it in two parts: offline evaluation before release, and monitoring after.
Offline evaluation. Build a set of tasks with a known end state, such as "the ticket is closed with the right tag". Run the agent against them and score the outcome, not the wording. Record steps, tools called and cost too. Include tasks where the right move is to stop and ask a person. Re-run the set when you change the prompt, model, a tool or its description.
Online monitoring. Alert on the patterns that mean the agent is off course:
| Signal | Why it matters | Typical alert |
|---|---|---|
| Steps per run rising | Why it matters: Looping or poor planning | Typical alert: Step-limit hits above your baseline |
| Same tool, same arguments, repeatedly | Why it matters: A stuck loop | Typical alert: Identical calls above a set count in one run |
| Tool errors and denied calls | Why it matters: Broken tools or an agent reaching for things it should not | Typical alert: Error rate by tool; any call blocked by a permission check |
| Spend per run and per tenant | Why it matters: Cascading cost | Typical alert: Run cost above your cap; daily tenant spend above a ceiling |
| Actions taken without approval where approval is required | Why it matters: A control failure | Typical alert: Any occurrence, treat as an incident |
OWASP publishes a Top 10 for Agentic Applications ("Version 2026", published 9 December 2025). Several of its entries describe things you can now see in your traces: ASI01 Agent Goal Hijack, ASI02 Tool Misuse and Exploitation, ASI03 Identity and Privilege Abuse and ASI08 Cascading Failures. The OWASP Top 10 for LLM Applications 2025 lists LLM06 Excessive Agency and LLM10 Unbounded Consumption. Logging the tool, the identity and the step count gives you evidence for those risks. That is mapping your monitoring to the OWASP lists, not a claim of compliance.
If your agents use MCP servers, remember that the specification recommends a human in the loop with the ability to deny tool invocations, and that clients must treat tool annotations as untrusted unless they come from trusted servers. Log the approvals and denials.
Which alerts are worth setting up first?
Start with six alerts that someone will act on:
- Cost per request, 95th percentile, above a threshold you set from a week of data.
- Daily spend per tenant above a ceiling.
- Error rate from the provider or your own validation, above a baseline.
- Retry or loop count above the limit.
- Latency, 95th percentile, for the full request.
- Quality proxy drop: format failures, refusals or thumbs-down rate above a baseline after a release.
Give each alert one named owner and a one-line runbook, such as "roll back the prompt version".
What does a one-week setup look like?
- Day 1. Wrap every model call in one function. Record the fields in the table above, a trace ID and a computed cost. Send them to your tracing tool or log store.
- Day 2. Add a prompt version string and a tenant ID to every call. Put model prices in a dated config table.
- Day 3. Add hard checks: schema validity, refusals, finish reason. Record them as scores on the trace.
- Day 4. Add user feedback capture and link it to the trace ID.
- Day 5. Build the golden set from real, anonymised cases and run it in CI on prompt or model changes.
- Day 6. Set the alerts and the step, retry and spend caps in code.
- Day 7. Write down redaction, sample size, retention and access rules for any stored text.
After that, review a week of traces and fix the costliest requests first. If you later weigh moving workloads onto your own infrastructure, the same traces give you the volume data to compare options. Our Own Your AI service covers that path.
Sources: OpenTelemetry semantic conventions for generative AI (github.com/open-telemetry/semantic-conventions-genai), status Development. Langfuse pricing, langfuse.com/pricing, checked 20 November 2026. OpenAI, Anthropic and Google API pricing pages, checked 20 November 2026. OWASP Top 10 for LLM Applications 2025 (genai.owasp.org/llm-top-10) and OWASP Top 10 for Agentic Applications for 2026 (genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026). Model Context Protocol specification, 2026-07-28, Tools. Digital Personal Data Protection Act, 2023 and DPDP Rules, 2025 (core duties from 13 May 2027). Last reviewed 20 November 2026.
FAQ
What is the minimum LLM observability setup for a small team?
One trace per user request, and one log row per model call with the model, input and output tokens, latency, status, prompt version, tenant ID and computed cost. Add a few cheap quality checks, such as output format validity, refusals and user feedback. Keep prompt text out of the logs by default. A team can set this up in about a week with open tooling.
Should we log full prompts and responses?
Not by default. Prompts and responses often hold personal data, customer secrets and credentials. Log metadata for every call, and capture text only for a defined sample, such as failed or thumbs-down calls. Redact first, set a retention period, restrict access, and treat the trace store as a data store that your data protection duties cover.
How do we track LLM cost per customer?
Put a tenant or customer ID on every trace and span, record input, output and cached tokens, and compute cost in your own wrapper using a dated table of model prices. Then sum cost by tenant per day and compare it with what the customer pays. Alert when a tenant passes a daily ceiling.
How do we monitor an AI agent in production?
Trace every step with the tool name, result status, identity used and stop reason. Alert on steps per run nearing the limit, repeated identical tool calls, tool errors and denied calls, spend per run, and any action taken without a required approval. Put step, retry and spend caps in code, and run an outcome-based test set before every prompt, model or tool change.
Why does our AI feature work in the demo but fail in production?
Demos use clean, short inputs and a lucky run. Production brings messy inputs, long histories, many tenants, provider errors and unexpected use. Common causes are missing timeouts and retries, growing context, wrong retrieval results, silent prompt or model changes and no output validation. Each one shows up as a signal you can log and a fix you can test.
Is the OpenTelemetry GenAI convention stable enough to use?
It is an emerging standard. Its status is Development, not stable, and its attribute and metric names can change. It is still worth following for attributes such as model, provider and token usage. Keep your own field names in a thin wrapper so a rename touches one file, and treat prompt and response content as opt-in.
