LLM Observability Cheatsheet: Tracing, Tokens, Cost and OpenTelemetry GenAI

An LLM feature in production fails in ways a normal service does not. The HTTP call returns 200 and the answer is wrong. Latency doubles because a prompt grew by 30K tokens. The monthly bill jumps because one agent started looping through a tool eight times instead of two. None of that shows up in a dashboard that only tracks request rate, errors and p95 latency.

LLM observability is the set of signals that make these failures visible: traces of every model call and tool call inside a request, token counts, cost, time to first token, finish reasons, and (carefully) the content itself. The good news is that you do not need a new stack for it. OpenTelemetry has a set of semantic conventions for generative AI that define the span names, attribute names and metrics, so an LLM call can live in the same trace as the database query and the HTTP request around it.

This cheatsheet covers those conventions as of release v1.38.0 (October 29, 2025), how to model LLM calls, tools and agents as spans, how to account for tokens and cost, which latencies to measure, how to sample and redact, and what the main tools do. It is about runtime telemetry, not quality evaluation.

1. What to observe

The first six are cheap, safe and should be on for every call. Content is the most useful for debugging and the most dangerous to store (section 9).

2. OpenTelemetry GenAI semantic conventions: status

The GenAI conventions are part of the OpenTelemetry semantic conventions repository. Every GenAI attribute, span and metric is marked Development (experimental). Names have changed between releases and may change again.

What changed recently, and why it matters if you read older blog posts or dashboards:

The spec includes a transition plan for this churn. Instrumentations that emitted v1.36.0 or earlier should not change their default output, and should support OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental to emit the latest experimental version instead. In practice: pin your instrumentation versions, check which convention version they emit, and expect to update dashboards when you upgrade.

3. Spans for LLM calls (inference)

Key attributes, with their requirement level in v1.38.0:

Two details worth noticing:

• Request model vs response model. You ask for an alias; the provider answers with a specific snapshot. Record both. When quality or latency shifts overnight, gen_ai.response.model is often the explanation.
• Finish reasons are a cheap, high-value signal. A rising share of length means truncated answers; a rise in content-filter stops means users or prompts changed.

4. Spans for tools and agents

An agent request is a tree. The conventions give you names for each node:

invoke_agent support-agent              (CLIENT or INTERNAL)
├── chat gpt-4o                          (CLIENT)  model decides to call a tool
├── execute_tool search_orders           (INTERNAL)
│   └── GET /orders/1234                 (regular HTTP client span)
├── chat gpt-4o                          (CLIENT)  model reads the result
└── execute_tool issue_refund            (INTERNAL)

What this buys you:

• Loops become visible. Count execute_tool spans per invoke_agent span. An agent that normally uses three tool calls and suddenly uses twelve is stuck.
• Tool latency vs model latency. Often the slow part of an "LLM request" is a tool, not the model.
• Tool errors in context. Invalid arguments, empty results and timeouts sit next to the model call that produced them, with the same gen_ai.tool.call.id.

5. Metrics

The conventions define histograms so you do not have to aggregate from spans:

All of them carry gen_ai.operation.name, gen_ai.provider.name, and the request and response model, so you can break down by model and provider. Do not put user IDs or conversation IDs on metrics; high-cardinality labels belong on spans.

6. Tokens and cost

Tokens are the unit of both cost and capacity. Record them from the provider's response (usage), not from your own tokenizer estimate.

As of v1.38.0 there is no standard attribute for cost, cached input tokens or reasoning tokens. Use your own namespace (for example app.llm.cost_usd, app.llm.cached_input_tokens) so they do not collide with future official names, and document them.

Cost per request is simple arithmetic:

cost = input_tokens  * input_price_per_token
     + output_tokens * output_price_per_token
     (- cached_input_tokens * discount, if your provider reports them)

Useful alerts: cost per request p95 by feature, tokens per agent run, and the ratio of input tokens to output tokens (a sudden jump in input usually means context or retrieval grew).

7. Latency: duration, TTFT, tokens per second

For streaming responses, total duration is the wrong number to optimize alone. Users perceive:

The conventions define TTFT as a server metric. For client-side TTFT (what your users experience through a provider API) there is no standard attribute in v1.38.0, so measure it in your client and record it as a custom attribute or metric. Always report latency per model and per output-length bucket; averages across very different requests hide everything.

8. A small Python example

Plain OpenTelemetry SDK, no vendor library. A fake streaming client stands in for a provider so the example runs as is (pip install opentelemetry-sdk). Prices are illustrative.

import time
from opentelemetry import metrics, trace
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.trace import SpanKind, Status, StatusCode

resource = Resource.create({"service.name": "support-bot"})
tp = TracerProvider(resource=resource)
tp.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(tp)
mp = MeterProvider(resource=resource,
                   metric_readers=[PeriodicExportingMetricReader(ConsoleMetricExporter())])
metrics.set_meter_provider(mp)

tracer = trace.get_tracer("support-bot.llm")
meter = metrics.get_meter("support-bot.llm")
token_usage = meter.create_histogram("gen_ai.client.token.usage", unit="{token}")
duration = meter.create_histogram("gen_ai.client.operation.duration", unit="s")

PRICES = {"example-model": {"input": 0.50, "output": 2.00}}  # USD per 1M tokens, illustrative


def fake_stream(prompt):
    """Stand-in for a provider streaming call: text chunks, then a usage record."""
    for word in ["Your", " refund", " is", " on", " its", " way."]:
        time.sleep(0.05)
        yield word
    yield {"input_tokens": 412, "output_tokens": 6, "response_model": "example-model-2025-10-01"}


def chat(model: str, prompt: str) -> str:
    base = {"gen_ai.operation.name": "chat",
            "gen_ai.provider.name": "openai",
            "gen_ai.request.model": model}
    start = time.perf_counter()
    with tracer.start_as_current_span(f"chat {model}", kind=SpanKind.CLIENT,
                                      attributes=base) as span:
        span.set_attribute("gen_ai.request.max_tokens", 256)
        parts, usage, ttft = [], {}, None
        try:
            for chunk in fake_stream(prompt):
                if isinstance(chunk, dict):
                    usage = chunk
                    continue
                if ttft is None:
                    ttft = time.perf_counter() - start
                    span.set_attribute("app.llm.time_to_first_token", ttft)  # custom
                parts.append(chunk)
        except Exception as exc:
            span.set_status(Status(StatusCode.ERROR))
            span.set_attribute("error.type", type(exc).__qualname__)
            raise
        span.set_attribute("gen_ai.response.model", usage["response_model"])
        span.set_attribute("gen_ai.usage.input_tokens", usage["input_tokens"])
        span.set_attribute("gen_ai.usage.output_tokens", usage["output_tokens"])
        span.set_attribute("gen_ai.response.finish_reasons", ["stop"])
        price = PRICES[model]
        cost = (usage["input_tokens"] * price["input"]
                + usage["output_tokens"] * price["output"]) / 1_000_000
        span.set_attribute("app.llm.cost_usd", round(cost, 8))  # custom
    duration.record(time.perf_counter() - start, base)
    token_usage.record(usage["input_tokens"], {**base, "gen_ai.token.type": "input"})
    token_usage.record(usage["output_tokens"], {**base, "gen_ai.token.type": "output"})
    return "".join(parts)


print(chat("example-model", "Where is my refund?"))
tp.shutdown()
mp.shutdown()

The console exporter prints a span named chat example-model with kind CLIENT, the gen_ai.* attributes, a TTFT of about 0.05 s and a cost of 0.000218 USD, followed by the two histograms. Swap the console exporters for OTLP exporters to send the same data to any OTel-compatible backend.

If you use the OpenAI Python SDK, the contrib package opentelemetry-instrumentation-openai-v2 produces these spans for you. Run with opentelemetry-instrument python main.py, and content capture stays off unless you set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true.

9. Content capture, PII and redaction

The conventions are explicit: instructions, user messages and model outputs are "considered sensitive and are often large in size", and instrumentations "SHOULD NOT capture them by default". They describe three patterns:

Practical rules:

• Redact before export, in the process or the Collector. The Collector's redaction processor can drop attributes not on an allow list (allow_all_keys: false) and mask values that match regular expressions (blocked_values, for example card numbers or emails). The transform processor can truncate long attribute values.
• Treat tool arguments and results as content. gen_ai.tool.call.arguments and gen_ai.tool.call.result are opt-in for the same reason: they often contain customer data.
• Separate retention. Keep metadata (tokens, latency, model, finish reason) for months; keep content for days, in a store with stricter access.
• Hash, do not store, identifiers you only need for grouping (user ID, email), unless you need to look them up.
• Never log secrets in prompts. If an API key or password can end up in a prompt, it will end up in a trace.

10. Sampling

LLM traces are large, and the interesting ones are rare. Plain head sampling (keep 1% decided at the start) throws away most of the failures you want.

The last row matters: compute cost and token metrics from every call, not from sampled traces, or your cost numbers will be wrong by the sampling rate.

11. Tools landscape

Facts from each project's own documentation. All of them accept or build on OpenTelemetry to some degree, which is the main reason to instrument with OTel conventions rather than a vendor SDK.

Note that OpenInference and the OTel GenAI conventions use different attribute names for similar concepts. If you mix instrumentations, check what each emits before building dashboards.

12. Checklist

• [ ] Every LLM call is a span with gen_ai.operation.name, gen_ai.provider.name, request and response model, and token usage.
• [ ] Cost is computed per call from a versioned price table and aggregated per request, feature and tenant.
• [ ] TTFT and total duration are tracked per model, with output length taken into account.
• [ ] Content capture is off by default; where it is on, it is redacted and stored with separate access and retention.
• [ ] Tail sampling keeps errors, slow and expensive traces; metrics are unsampled.

Why this matters

• LLM failures are silent at the HTTP level. Tokens, finish reasons and tool spans make them visible.
• Cost is a production metric. Without per-call token accounting, the bill is the first alert.
• Agents need traces, not logs. A run is a tree of model and tool calls; only a trace shows where it looped or stalled.
• Standard names keep you portable. OTel GenAI conventions let you switch backends without re-instrumenting, even while they are still experimental.
• Content is both the best debugging signal and the biggest risk. Decide deliberately where it goes.

Final Thoughts

Start with metadata: one span per model call with the required attributes, token counts and a cost attribute, and tool spans under the same trace. That alone answers most production questions: what is slow, what is expensive, what changed. Add content capture only where you can store it safely, and tail sampling once volume makes it necessary.

The GenAI conventions are still marked as development and will keep moving. Pin versions, read the changelog when you upgrade, and keep your own additions in your own namespace. The shape (spans for calls, tools and agents; histograms for tokens and duration) is already stable enough to build on.

Sources

• OpenTelemetry, Semantic conventions for generative AI systems and the v1.38.0 GenAI docs: Development status, OTEL_SEMCONV_STABILITY_OPT_IN transition plan.
• OpenTelemetry, GenAI client spans, v1.38.0: span names and kinds, attributes and requirement levels, execute tool span, content capture patterns.
• OpenTelemetry, GenAI agent spans, v1.38.0: create_agent, invoke_agent, agent attributes.
• OpenTelemetry, GenAI metrics, v1.38.0: token usage and duration histograms, bucket boundaries, server TTFT and time per output token.
• OpenTelemetry, semantic-conventions releases: v1.37.0 (August 25, 2025) and v1.38.0 (October 29, 2025) GenAI changes.
• Drew Robbins and Liudmila Molkova, OpenTelemetry for Generative AI (OpenTelemetry blog, December 5, 2024): OpenAI instrumentation, opentelemetry-instrument, OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT.
• OpenTelemetry Collector contrib, redaction processor and tail sampling processor.
• Langfuse, Python SDK v3 generally available (June 5, 2025) and OpenTelemetry docs: OTel-based SDK, OTLP endpoint.
• Arize, Phoenix repository: license, built on OpenTelemetry and OpenInference.
• Arize, OpenInference repository: conventions complementary to OpenTelemetry.
• Traceloop, OpenLLMetry repository: Apache 2.0, extensions built on OpenTelemetry, Traceloop.init().