← All news and guides
Engineering

LLM Observability Setup: Traces, Costs and Quality Signals for Small Teams

What a small team should log, score and alert on for an LLM feature: traces, cost per request, quality checks, agent monitoring and a one-week setup plan.

Mono Team ·

A wall of small analogue gauges and indicator lamps in a dim control room, with a few lit amber, seen in soft light.

Short answer: a small team needs one trace per user request, one log row per model call, a cost figure on every row, and three or four cheap quality signals. That is enough. Log the model, token counts, latency, status, tool calls, prompt version and a tenant ID. Add cost, cache hits and retries. Score a sample of answers with user feedback, format checks and a small test set. Alert on cost per request, error rate, retries and loop length. Keep prompt and response text out of the logs unless you have a reason, a retention limit and redaction. You can do all of this with open tooling in about a week.

This guide is for small teams that ship an AI feature and cannot yet answer "why did that cost so much?" or "is it getting worse?". It pairs with our guide to why LLM bills grow faster than usage and the AI spend audit. It is engineering guidance, not legal advice.

What is LLM observability, and why is normal monitoring not enough?

Normal monitoring tells you a request returned 200 in 900 ms. An LLM feature can do that and still be wrong or far dearer than yesterday. The same input can give a different answer, one user action can fan out into several model calls and tool calls, and cost depends on tokens, not requests.

LLM observability adds three things on top of ordinary logs and metrics:

  • Traces that show every step behind one user request: retrieval, each model call, each tool call, and the final answer.
  • Cost and usage per call, so you can see which feature, tenant or prompt version is driving spend.
  • Quality signals, so you can tell "the answer was wrong" from "the system was slow".

What should you log for every LLM call?

A tidy workbench with a row of small labelled sample trays lined up under a lamp, each tray holding a few neutral components, soft warm light, shallow depth of field.

Start with the fields below. Most of them are metadata, so they carry little privacy risk. Where the OpenTelemetry GenAI semantic conventions already name a field, we show the attribute. That convention set is an emerging standard: its status is Development, not stable, and the project has moved to its own repository (open-telemetry/semantic-conventions-genai). Pin the version you use and expect renames.

FieldWhy you need itOpenTelemetry attribute (emerging)
Trace ID and parent spanWhy you need it: Ties a model call to the user request and to the steps around itOpenTelemetry attribute (emerging): Standard trace context
Operation type (chat, embeddings, tool)Why you need it: Lets you separate generation cost from retrieval costOpenTelemetry attribute (emerging): `gen_ai.operation.name`
Provider and modelWhy you need it: Needed to price the call and to spot a model changeOpenTelemetry attribute (emerging): `gen_ai.provider.name`, `gen_ai.request.model`
Input and output tokensWhy you need it: The base for every cost figureOpenTelemetry attribute (emerging): `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`
Cached input tokensWhy you need it: Shows whether prompt caching is workingOpenTelemetry attribute (emerging): `gen_ai.usage.cache_read.input_tokens`
Latency and statusWhy you need it: Finds slow steps and failuresOpenTelemetry attribute (emerging): Span duration and status
Prompt version and feature nameWhy you need it: Links a cost or quality change to a releaseOpenTelemetry attribute (emerging): Your own attribute
Tenant or customer ID, hashed or internalWhy you need it: Finds the heavy users and per-customer marginOpenTelemetry attribute (emerging): Your own attribute
Retry count and loop step numberWhy you need it: Shows hidden repeat spendingOpenTelemetry attribute (emerging): Your own attribute
Computed cost in rupees or dollarsWhy you need it: Lets finance and engineering read the same numberOpenTelemetry attribute (emerging): Your own attribute

Do not depend on the OpenTelemetry metric names for now, because they are changing. Keep your own names in a thin wrapper.

Should you store prompts and responses?

Only with a reason. Prompt and response text is where personal data, customer secrets and credentials end up. The OpenTelemetry conventions treat message content (`gen_ai.input.messages` and `gen_ai.output.messages`) as opt-in for this reason. Treat your own logging the same way.

  • Default to metadata only. Tokens, timings, IDs and scores answer most cost and reliability questions.
  • Capture text for a small sample, such as failed calls, thumbs-down calls and a random slice.
  • Redact before you store. Strip names, phone numbers, emails, PAN and Aadhaar numbers, and API keys. See our guide to redacting PII before LLM calls. Redaction reduces leakage. Test its recall on your own data before you rely on it.
  • Set a retention period and enforce it. A trace store is a data store. If it holds personal data, it needs the same access control, erasure and log retention thinking as the rest of your system.
  • Know where it sits. A hosted tracing tool that stores personal data is usually acting as your Data Processor under the DPDP Act. From 13 May 2027 you, as the Data Fiduciary, stay responsible for that data and need a valid contract with the vendor (s.8(1) and s.8(2)). How this applies to your setup depends on the facts, so check it with counsel.

Our DPDP-aware AI architecture guide covers where to place redaction and logging in the request path.

Which tool should a small team use?

You have three sensible routes. None of them is wrong. Pick the one that fits your data rules and your appetite for running software.

RouteGood forWatch out for
Hosted LLM tracing toolGood for: Fast start, a UI for traces and scores, no servers to runWatch out for: Text may leave your network. Check regions and data terms
Self-hosted open-source tracing toolGood for: Data stays on your infrastructure; no per-unit feeWatch out for: You run and patch it, and you pay for the database and storage
OpenTelemetry into the stack you already haveGood for: One place for app and LLM telemetry; no new vendorWatch out for: Conventions are still moving, and you build the cost and quality views yourself

To give one concrete public price point: Langfuse Cloud lists a free Hobby plan (50k units a month, 30 days of data access), a Core plan at $29 a month and a Pro plan at $199 a month, with Enterprise at $2,499 a month. Core and Pro include 100k units and then charge $8 per 100k. Data access is 90 days on Core and 3 years on Pro. Its cloud regions are US, EU and Japan, with no India region. Self-hosting it is free. Those are public list prices checked on 20 November 2026 (https://langfuse.com/pricing) and they change. This is an example, not a recommendation.

If customers require data to stay in India, a hosted tool in another region may fail that requirement for prompt text. Self-host it, or send metadata only. If you run models on your own servers, keep the tracing there too. See Own Your AI.

How do you trace a multi-step request or an agent?

Create one trace per user request. Make each model call, retrieval and tool call a child span. Pass the trace ID through queues and background jobs.

For agents, add these to every span:

  • Step number within the run, and the run's step limit.
  • Tool name, arguments hash and result status. Store the arguments themselves only under the text rules above.
  • Stop reason: finished, step limit hit, error, timeout, or handed to a human.
  • Identity used by the tool call.
  • Approval event, if a human confirmed or rejected the action.

Two limits belong in code, not in a dashboard: caps on steps and retries per request, and a spend ceiling per request and per tenant per day. The logs then show how often the caps fire.

What cost signals should you watch?

Cost per model call is easy. Cost per user outcome matters more: sum the cost of every span in a trace and divide by completed outcomes, such as resolved tickets.

Watch these, in this order:

  • Cost per request, as a distribution. Look at the median and the 95th percentile. A few runaway requests hide inside an average.
  • Input tokens per request by conversation turn. Rising input tokens mean history or retrieved context is growing.
  • Output tokens. At the list prices we checked on 20 November 2026 for OpenAI and Anthropic, output costs five times input per token. Long answers and hidden reasoning tokens add up.
  • Cache hit rate. Divide cached input tokens by total input tokens. Cached reads are billed at a fraction of the normal input price (our AI spend audit lists the current multipliers and a worked example). A low hit rate on a long, repeated prompt means something near the top of the prompt changes on every call.
  • Calls per user action and retries, since loops and retries repeat full prompts.
  • Cost per tenant. The customers whose usage costs more than they pay.
  • Batch share. Work that is not live can often go through a batch API. Anthropic's and OpenAI's lists show Batch at 50% off.

Put unit prices in a small config table with an effective date, and compute cost in your wrapper. Prices change: Google's Gemini 4 Carbon list price is scheduled to rise from $2 to $4 per million input tokens on 1 December 2026. A dated table lets you re-cost old traces correctly.

What quality signals can a small team collect cheaply?

Hands sorting a small pile of identical wooden blocks into two shallow trays on a plain table, soft daylight from the side, no faces, shallow depth of field.

Layer these, cheapest first:

  • Hard checks on every response. Does the output parse as the JSON you asked for? Did the model refuse? Did it hit the length limit? Did it cite a document that was not retrieved? These are free and catch real failures.
  • User signals. Thumbs up or down, a regenerate click, a copy click, an edit before sending, a follow-up that repeats the question. Treat them as signals, not truth.
  • Retrieval checks for RAG: was any document retrieved, how many, and did the answer use it? An answer with no supporting source is a risk flag.
  • A golden set. A few dozen real, anonymised cases with known good answers, run on every prompt or model change. Add each production failure to it once you have fixed it.
  • An LLM judge on a sample. A second model scores answers against a rubric. Check the judge against human labels on a few dozen cases before trusting it, and re-check when you change the judge.
  • Human review of a small weekly sample, including the worst-scoring answers.

Store each score against the trace ID, prompt version and model. Then "quality fell" becomes answerable: after which release, for which tenant, on which questions.

Why does an AI feature work in the demo and fail in production?

Demos use clean inputs, short contexts, one user and a lucky run. Production has messy inputs, long histories, many tenants, provider outages and unexpected use. You can observe most of that gap, then close it.

GapWhat you see in productionSignal to addFix to try
Inputs differ from the demoWhat you see in production: Odd formats, other languages, very long text, empty fieldsSignal to add: Input length and language distributionFix to try: Validate and normalise inputs; add real cases to the golden set
No timeout or retry designWhat you see in production: Hung requests, duplicate actionsSignal to add: Latency percentiles, retry count, timeout rateFix to try: Timeouts, bounded retries with back-off, idempotency keys for actions
Context grows over a sessionWhat you see in production: Slow, costly, then confused answersSignal to add: Input tokens by turnFix to try: Summarise or trim history; cap context
Retrieval returns the wrong thingWhat you see in production: Confident answers from the wrong documentSignal to add: Retrieved count, source IDs, empty-retrieval rateFix to try: Fix chunking and permissions; answer "I do not know" when nothing is found
Prompt or model changed silentlyWhat you see in production: Behaviour shifts with no code releaseSignal to add: Prompt version and model on every spanFix to try: Pin model versions; run the golden set before changing
No fallback when the model is wrongWhat you see in production: Bad answers reach the user or the databaseSignal to add: Validation failure rate, human-override rateFix to try: Output validation, a human check for risky actions

Our AI product development team does this kind of hardening work.

How do you evaluate and monitor an AI agent?

An agent makes decisions between the user and the result, so monitor decisions and not only answers. Do it in two parts: offline evaluation before release, and monitoring after.

Offline evaluation. Build a set of tasks with a known end state, such as "the ticket is closed with the right tag". Run the agent against them and score the outcome, not the wording. Record steps, tools called and cost too. Include tasks where the right move is to stop and ask a person. Re-run the set when you change the prompt, model, a tool or its description.

Online monitoring. Alert on the patterns that mean the agent is off course:

SignalWhy it mattersTypical alert
Steps per run risingWhy it matters: Looping or poor planningTypical alert: Step-limit hits above your baseline
Same tool, same arguments, repeatedlyWhy it matters: A stuck loopTypical alert: Identical calls above a set count in one run
Tool errors and denied callsWhy it matters: Broken tools or an agent reaching for things it should notTypical alert: Error rate by tool; any call blocked by a permission check
Spend per run and per tenantWhy it matters: Cascading costTypical alert: Run cost above your cap; daily tenant spend above a ceiling
Actions taken without approval where approval is requiredWhy it matters: A control failureTypical alert: Any occurrence, treat as an incident

OWASP publishes a Top 10 for Agentic Applications ("Version 2026", published 9 December 2025). Several of its entries describe things you can now see in your traces: ASI01 Agent Goal Hijack, ASI02 Tool Misuse and Exploitation, ASI03 Identity and Privilege Abuse and ASI08 Cascading Failures. The OWASP Top 10 for LLM Applications 2025 lists LLM06 Excessive Agency and LLM10 Unbounded Consumption. Logging the tool, the identity and the step count gives you evidence for those risks. That is mapping your monitoring to the OWASP lists, not a claim of compliance.

If your agents use MCP servers, remember that the specification recommends a human in the loop with the ability to deny tool invocations, and that clients must treat tool annotations as untrusted unless they come from trusted servers. Log the approvals and denials.

Which alerts are worth setting up first?

Start with six alerts that someone will act on:

  • Cost per request, 95th percentile, above a threshold you set from a week of data.
  • Daily spend per tenant above a ceiling.
  • Error rate from the provider or your own validation, above a baseline.
  • Retry or loop count above the limit.
  • Latency, 95th percentile, for the full request.
  • Quality proxy drop: format failures, refusals or thumbs-down rate above a baseline after a release.

Give each alert one named owner and a one-line runbook, such as "roll back the prompt version".

What does a one-week setup look like?

  • Day 1. Wrap every model call in one function. Record the fields in the table above, a trace ID and a computed cost. Send them to your tracing tool or log store.
  • Day 2. Add a prompt version string and a tenant ID to every call. Put model prices in a dated config table.
  • Day 3. Add hard checks: schema validity, refusals, finish reason. Record them as scores on the trace.
  • Day 4. Add user feedback capture and link it to the trace ID.
  • Day 5. Build the golden set from real, anonymised cases and run it in CI on prompt or model changes.
  • Day 6. Set the alerts and the step, retry and spend caps in code.
  • Day 7. Write down redaction, sample size, retention and access rules for any stored text.

After that, review a week of traces and fix the costliest requests first. If you later weigh moving workloads onto your own infrastructure, the same traces give you the volume data to compare options. Our Own Your AI service covers that path.

Sources: OpenTelemetry semantic conventions for generative AI (github.com/open-telemetry/semantic-conventions-genai), status Development. Langfuse pricing, langfuse.com/pricing, checked 20 November 2026. OpenAI, Anthropic and Google API pricing pages, checked 20 November 2026. OWASP Top 10 for LLM Applications 2025 (genai.owasp.org/llm-top-10) and OWASP Top 10 for Agentic Applications for 2026 (genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026). Model Context Protocol specification, 2026-07-28, Tools. Digital Personal Data Protection Act, 2023 and DPDP Rules, 2025 (core duties from 13 May 2027). Last reviewed 20 November 2026.

FAQ

What is the minimum LLM observability setup for a small team?

One trace per user request, and one log row per model call with the model, input and output tokens, latency, status, prompt version, tenant ID and computed cost. Add a few cheap quality checks, such as output format validity, refusals and user feedback. Keep prompt text out of the logs by default. A team can set this up in about a week with open tooling.

Should we log full prompts and responses?

Not by default. Prompts and responses often hold personal data, customer secrets and credentials. Log metadata for every call, and capture text only for a defined sample, such as failed or thumbs-down calls. Redact first, set a retention period, restrict access, and treat the trace store as a data store that your data protection duties cover.

How do we track LLM cost per customer?

Put a tenant or customer ID on every trace and span, record input, output and cached tokens, and compute cost in your own wrapper using a dated table of model prices. Then sum cost by tenant per day and compare it with what the customer pays. Alert when a tenant passes a daily ceiling.

How do we monitor an AI agent in production?

Trace every step with the tool name, result status, identity used and stop reason. Alert on steps per run nearing the limit, repeated identical tool calls, tool errors and denied calls, spend per run, and any action taken without a required approval. Put step, retry and spend caps in code, and run an outcome-based test set before every prompt, model or tool change.

Why does our AI feature work in the demo but fail in production?

Demos use clean, short inputs and a lucky run. Production brings messy inputs, long histories, many tenants, provider errors and unexpected use. Common causes are missing timeouts and retries, growing context, wrong retrieval results, silent prompt or model changes and no output validation. Each one shows up as a signal you can log and a fix you can test.

Is the OpenTelemetry GenAI convention stable enough to use?

It is an emerging standard. Its status is Development, not stable, and its attribute and metric names can change. It is still worth following for attributes such as model, provider and token usage. Keep your own field names in a thin wrapper so a rename touches one file, and treat prompt and response content as opt-in.

Need help making your AI feature reliable and measurable?