feat: Support for OpenTelemetry GenAI semantic conventions

[vasspilka]

Summary

Addresses #467 by adding OpenTelemetry GenAI telemetry to LangChain, following the OpenTelemetry GenAI semantic conventions (v1.40+). LangChain already emitted :telemetry events; this PR layers an opt-in OpenTelemetry integration on top that turns those events into spans and metrics, with privacy-safe defaults — and fixes the telemetry surface so token usage actually reaches those spans across every provider and streaming.

Spans carry a rich subset of the convention: request parameters, token usage (including cache/reasoning breakdowns), finish reasons, streaming time-to-first-token, tool metadata, server.*, and conversation grouping — all emitted best-effort, only when the model actually sets them.

For risks note breaking changes below

OpenTelemetry is an optional dependency — if :opentelemetry_api isn't loaded, the integration module isn't even compiled and there's zero overhead.

What's added

New LangChain.OpenTelemetry integration (lib/open_telemetry/)

Module	Responsibility
OpenTelemetry	Entry point — setup/1, teardown/0, without_tracing/1. Attaches span + metrics handlers.
OpenTelemetry.Config	Capture/feature config struct (all message capture defaults to false).
OpenTelemetry.SpanHandler	Turns :telemetry lifecycle events into spans with parent/child hierarchy. Records streaming first-token as a span attribute + event. Traps its own exceptions so a bad payload can't detach tracing VM-wide.
OpenTelemetry.MetricsHandler	Re-emits operation-duration, token-usage, and time-to-first-token metric events.
OpenTelemetry.Attributes	Builds GenAI-convention span attributes — request params, usage (incl. cache/reasoning), finish reasons, server.*, tool metadata, conversation.id, and optional Langfuse context.
OpenTelemetry.MessageSerializer	Serializes messages to JSON for gen_ai.*.messages attributes.
OpenTelemetry.ProviderMapping	Maps LangChain provider strings to GenAI provider names.

Span attributes (GenAI semantic conventions subset)

Emitted best-effort — a value appears only when the model actually sets it.

• Operation / identity: gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.agent.name, gen_ai.conversation.id
• Request parameters: gen_ai.request.temperature, .max_tokens, .top_p, .top_k, .frequency_penalty, .presence_penalty, .seed, .choice.count, .stream, .stop_sequences, .reasoning.level; plus gen_ai.output.type ("json" for structured-output requests, else "text")
• Server: server.address, server.port (derived from the model's request endpoint)
• Usage: gen_ai.usage.input_tokens, .output_tokens; best-effort gen_ai.usage.cache_read.input_tokens, .cache_creation.input_tokens, .reasoning.output_tokens
• Response: gen_ai.response.finish_reasons (reconstructed from the normalized message status)
• Streaming: gen_ai.response.time_to_first_token (seconds) + a gen_ai.first_token span event on the LLM span, for providers that stream through the shared path
• Tools: gen_ai.tool.name, .call.id, .type, .description

Telemetry surface

• LangChain.Telemetry gained documented metadata (auto call_id correlation, provider, custom_context, token usage, last_message, request_options, output_type, endpoint) and explicit privacy guarantees.
• Every in-tree chat model now reports its provider: 13 implement the new ChatModel.provider/0 callback, and ChatReqLLM derives the provider per model spec at call time. ChatModel.provider/1 falls back to deriving it from the module name for custom models that don't implement the optional callback.
• LLM-call telemetry is funneled through a single chokepoint, ChatModel.llm_telemetry_span/3, which injects request_options (standard model params), output_type, and endpoint, and centralizes token-usage extraction so no provider can silently drop it.
• Streaming time-to-first-token is captured via a new [:langchain, :llm, :stream, :first_token] event, primed by llm_telemetry_span/3 and emitted on the first delta through the shared streaming path.
• LLMChain wraps chain execution, run_until_tool_used, and tool calls in spans, aggregating token usage across all assistant messages of the run (multi-turn/tool-calling safe; scoped to exchanged_messages so persisted history isn't re-counted).
• LLMChain's built-in tool executor propagates the OpenTelemetry context into async: true tool Tasks automatically, so async tool spans nest under the chain span. The propagation is a runtime no-op unless the OTel deps are installed.

Span hierarchy

invoke_agent llm_chain   [internal]
  └─ chat <model>         [client]
      └─ execute_tool <name> [internal]

Bug fixes included

• ChatMistralAI token usage — a real product bug, not just a telemetry gap. Mistral fired the :on_llm_token_usage callback but, unlike every other chat model, never persisted usage onto the returned message. So TokenUsage.get/1 returned nil for every non-streaming Mistral response — any caller reading usage/cost off a Mistral message got nothing, regardless of telemetry. Usage is now attached via TokenUsage.set/2 (msg.metadata.usage).
• Streaming token usage now reaches the :stop telemetry and OTel spans. The shared ChatModel.token_usage_from_result/1 (:enrich_stop) only matched %Message{} results, so streaming calls — which return %MessageDelta{} with usage on the final delta — reported nil usage at the call boundary. It now handles single/list %MessageDelta{} and list results. (TokenUsage.get/1 on the accumulated streaming message was already correct — deltas accumulate usage and to_message/1 preserves it — so this fixes the telemetry surface, not get/1.)
• Mistral batches streamed deltas as a list of lists. Some providers (Mistral) return streamed deltas batched as a list-of-lists rather than a flat list, so the usage-bearing final delta sat one level deeper than the scan looked — silently dropping token usage from the :stop event and OTel span. token_usage_from_result/1 now List.flatten/1s before scanning (a no-op on already-flat results, safe for every provider's shape).
• Failed operations are visible to latency metrics. [:langchain, *, :exception] events now carry a duration measurement and an error.type attribute; spans close on exit/throw, not just raise.

⚠️ Breaking changes & upgrade notes

These only affect code that attaches to LangChain's :telemetry events or implements a custom chat model. Normal chain/LLM usage is unaffected, and no public function signatures changed.

Breaking

1. Chain telemetry metadata key tool_count → tools_count.
   Events [:langchain, :chain, :execute, :start | :stop | :exception] now
   expose :tools_count (plural) instead of :tool_count.

   Why: the LLM-call events ([:langchain, :llm, :call, *]) already emit
   this field as :tools_count across all in-tree chat models, while the chain
   events were the lone holdout emitting :tool_count. The same conceptual
   field was therefore named two different ways depending on which event you
   attached to. This renames the two chain-event sites to match the key already
   used by every LLM-call event — the lower-breakage direction (2 call sites
   aligned to the existing 13, rather than renaming the shipped :tools_count
   on the LLM events). It is not tied to any OpenTelemetry attribute —
   tools_count is passthrough telemetry metadata and is not emitted as a span
   attribute.

   Action: if you read metadata.tool_count in a handler attached to
   [:langchain, :chain, :execute, *], rename it to tools_count. Handlers on
   LLM-call events are unaffected (they already used tools_count).

2. ChatGrok message.metadata.usage type changed: raw API map → %TokenUsage{}.
   Grok previously stored the raw provider map (%{"prompt_tokens" => …, "completion_tokens" => …}) on message.metadata.usage. It now stores a %LangChain.TokenUsage{} struct (input, output, raw), matching every other provider. Action: code reading msg.metadata.usage["prompt_tokens"] off a Grok message must switch to msg.metadata.usage.input / .output (or TokenUsage.get/1). Only affects Grok; other providers were already %TokenUsage{}.

3. Streaming/non-streaming response event names normalized.
   The per-model response events were emitted under a malformed keyword-in-path form: [:langchain, :llm, :response, streaming: true] (i.e. {:streaming, true}) and [:langchain, :llm, :response, streaming: false]. They are now [:langchain, :llm, :response, :streaming] and [:langchain, :llm, :response, :non_streaming] — matching what the moduledoc always documented (affects ChatAnthropic, ChatBumblebee, ChatGoogleAI). Action: a handler attached to the old streaming: true/false form stops firing; re-attach to the :streaming / :non_streaming names.

Behavior changes worth noting (non-breaking, additive)

4. custom_context is now included in chain and tool telemetry metadata.
   [:langchain, :chain, :execute, *] and [:langchain, :tool, :call, *] events now carry the chain's custom_context. If your custom_context holds sensitive data and you fan telemetry out to third parties, be aware it's now on those events. It is intentionally not included on LLM-level events (correlate via call_id). Existing handlers that ignore unknown keys are unaffected.

5. Mistral and Perplexity messages now carry %TokenUsage{} in metadata.
   ChatMistralAI and ChatPerplexity fired the :on_llm_token_usage callback but previously left msg.metadata.usage unset. Both now persist it via TokenUsage.set/2. Additive — nothing that read those messages breaks, but usage is now present where it was nil. (This is the Mistral product-bug fix above, and Perplexity had the same gap.)

6. Every LangChain telemetry event now carries a :call_id.
   Telemetry.span/4 and start_event/2 inject a :call_id (a UUID, via Map.put_new, so you can supply your own to override). All existing chain/tool/LLM/message-process events gain this key. Additive.

7. [:langchain, *, :exception] events now fire on throw/exit too, and carry a duration.
   Previously exception events carried no duration, and exits/timeouts/throws produced no event at all (leaking any span a consumer opened on :start). Now every failure emits :exception with a duration and an error.type. Note for handlers: exit/throw events set metadata.kind to :exit/:throw and metadata.error to nil — code that assumes kind == :error or a non-nil error should be relaxed.

8. ChatGrok now emits the full LLM-call telemetry suite it never emitted before.
   Grok's call/3 now routes through ChatModel.llm_telemetry_span/3, so it emits [:langchain, :llm, :call, :start|:stop|:exception], :prompt, :response, and :response, :non_streaming like every other provider. Existing Grok users get brand-new events; any handler on [:langchain, :llm, :call, *] now fires for Grok. Additive.

9. New LLM-call telemetry metadata: request_options, output_type, endpoint.
   [:langchain, :llm, :call, *] events now carry the model's standard request parameters (request_options), the requested output type, and the endpoint (when the model exposes one). Additive; existing handlers ignoring unknown keys are unaffected.

10. New [:langchain, :llm, :stream, :first_token] telemetry event.
    Emitted once per streaming LLM call, on the first delta, with a duration measurement (time-to-first-token). Additive.

11. New optional ChatModel.provider/0 callback.
    Declared with @optional_callbacks, so existing custom ChatModel implementations continue to compile and run unchanged — they simply get their provider derived from the module name via ChatModel.provider/1. Implement provider/0 to override.

12. Telemetry.span/3 is now Telemetry.span/4 (added an optional opts \\ [] for :enrich_stop). Backward-compatible — existing span/3 calls keep working unchanged.

13. New (optional) dependencies.
    :opentelemetry_api is added as optional: true; :opentelemetry and :opentelemetry_exporter are only: :test. Applications that want the integration must add the OTel deps themselves (see docs). Applications that don't are unaffected and pull in nothing new.

Privacy model

Message/tool content is never attached to lifecycle spans by default. Each capture is opt-in via setup/1:

Option	Default
capture_input_messages	false
capture_output_messages	false
capture_tool_arguments	false
capture_tool_results	false
enable_metrics	true

Note on metrics: enable_metrics: true does not record OTel histograms directly — it re-emits three intermediary events:

• [:langchain, :otel, :operation, :duration] — %{duration_s}
• [:langchain, :otel, :token, :usage] — %{tokens}, tagged gen_ai.token.type ("input"/"output")
• [:langchain, :otel, :operation, :time_to_first_token] — %{duration_s}, once per streaming call

Attach a consumer (Telemetry.Metrics + reporter, PromEx, …) to record them. Documented prominently in the guide and MetricsHandler moduledoc.

Grouping traces into conversations

LangChain has no first-class conversation id, so gen_ai.conversation.id (used by backends to group a multi-turn session) is read from the chain's custom_context — set :conversation_id (or reuse :langfuse_session_id) and it lands on the root chain span.

Usage

# application startup
LangChain.OpenTelemetry.setup()

# or, opting into content capture where PII is acceptable:
LangChain.OpenTelemetry.setup(capture_input_messages: true)

Documentation

• New Observability guide (guides/observability.md) — telemetry event catalog, OTel setup, span/metric mappings, metrics wiring, Langfuse integration, conversation grouping, async-tool context propagation, and the full config table. Rendered on HexDocs under Guides.
• New runnable Livebook notebooks/observability_tutorial.livemd — a hands-on tour of both layers (raw telemetry → OTel spans → Langfuse → metrics). Rendered under Notebooks.
• README — new ## Observability section with a minimal setup snippet and links.
• usage-rules.md — new Observability section so AI coding agents consuming LangChain get the telemetry rules (including the tools_count naming gotcha and PII-off-by-default).

(Breaking changes and upgrade notes are captured in this PR description rather than the CHANGELOG, which isn't edited manually.)

Followups & known limitations

Deferred or best-effort by design — none block this PR, but they're worth tracking:

• gen_ai.response.model reflects the requested model, not the returned one. When a provider echoes a more specific id (gpt-4o → gpt-4o-2024-08-06), that specific id isn't captured yet. Use gen_ai.request.model as source of truth.
• gen_ai.request.* is a best-effort subset. Parameters are read by conventional struct field name; a param under a non-standard field name is silently omitted.
• gen_ai.response.finish_reasons is best-effort, reconstructed from the normalized Message.status (stop / length / tool_calls / cancelled), not from the provider's raw finish reason.
• Cache/reasoning token counts are best-effort, only for providers that report them in usage.
• Time-to-first-token only covers the shared streaming decode path. It relies on process-dictionary markers set by llm_telemetry_span/3; streaming outside a telemetry span degrades to a no-op.
• provider/1 module-name fallback can't recover acronym/multi-word canonical names (ChatOpenAIResponses → "open_ai_responses"). Mitigated because every in-tree model implements provider/0; custom models should too.
• without_tracing/1 and async-tool context propagation are process-scoped. The chain auto-propagates OTel context only for its own built-in async-tool executor; processes you spawn yourself must re-attach context manually.
• Reserved-but-unemitted event families (:memory, :retriever) are kept as naming placeholders with no call sites yet — room for future instrumentation.

Dependencies

• :opentelemetry_api ~> 1.4 (optional)
• :opentelemetry ~> 1.5, :opentelemetry_exporter ~> 1.8 (test only)

Tests

~4,000 lines of new tests across test/open_telemetry/ (span handler, attributes, metrics handler, message serializer, config, provider mapping, without-tracing, async-tool context) and expanded test/chat_models/telemetry_test.exs, chat_model_test.exs, plus per-provider telemetry tests (Grok, Mistral, Perplexity, ReqLLM). The observability suite runs 207 tests, 0 failures; MetricsHandler, without_tracing/1, and async-tool context propagation are all covered (previously flagged as untested).

[brainlid]

This never left the draft state and appears to have been abandoned. Closing.

[vasspilka]

Hey @brainlid the PR works as is. We have been using it in Teacherspace in production for around 3 months now.
It's not ideal but solves the basic usecase of gathering the telemetry data

The reason it remained a draft is because and was waiting for your thoughts around GenAI SemConv support so we can better coordinate the change.

[brainlid]

Thanks @vasspilka for the update. I didn't realize you were waiting on me. I was waiting on the PR leaving draft status! 😆

Specifically, what would you like from me? Are you saying it's ready for a review and that's what you want? Or are you proposing something specific?

[vasspilka]

@brainlid Thanks for the quick reply, let me refresh my memory this work and I'll come back to you during the week.

Overall it's a net improvement. But could be a breaking changes if people relied on matching Telemetry [streaming: true]

[vasspilka]

@brainlid Sorry for the delay. I found that there were some issues, please give me a day or two and it will be finished

[vasspilka]

Hey @brainlid

I actually was not right when I said we've have this running in production. It was actually only our staging environment that had this live.

Upon further QA it worked well but was not a complete seamless solution. I've been working on getting full support on the OpenTelemetry GenAI semantic conventions specification.

Then will do a final round of testing. Note that the PR has grown considerably but the vast majority is tests and documentation.

[vasspilka]

@brainlid This is now ready for review.

Note the breaking changes (only for low level consumers of old telemetry). Most notably tool_count → tools_count

Also if someone would be to use ReqLLM as a provider they would get double traces. This is something we can address in a follow up or I can patch this PR.

Please fire away any questions or clarifications :)

Cheers

[brainlid]

@vasspilka, what are you thinking about preventing the double traces when using ChatReqLLM? I'm fine with a follow-up for it. Have you got a plan in mind?

[brainlid]

Thanks @vasspilka! It was a beast to review. But it adds a lot of value. Thanks for the guides, docs, and notebook. I can tell a lot of time was put into this. Even with a coding agent, there was still a lot of time and effort.
❤️💛💙💜

[brainlid]

@vasspilka released as v0.9.0