feat(codecs): support encoding native Vector metrics in OTLP serializer#25738
feat(codecs): support encoding native Vector metrics in OTLP serializer#25738petere-datadog wants to merge 2 commits into
Conversation
Adds metric -> OTLP conversion to the OTLP codec's serializer, covering Counter, Gauge, AggregatedHistogram, and AggregatedSummary values via direct prost encoding. Unsupported metric types (Set, Distribution, Sketch) return an error naming the variant. The metric -> OTLP -> metric round trip is deterministic; the reverse direction remains lossy by design (e.g. ExponentialHistogram flattens to AggregatedHistogram).
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1e9b90ea1a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| let bucket_counts: Vec<u64> = buckets.iter().map(|bucket| bucket.count).collect(); | ||
| let explicit_bounds: Vec<f64> = buckets | ||
| .iter() | ||
| .take(buckets.len().saturating_sub(1)) |
There was a problem hiding this comment.
For native AggregatedHistograms that do not already carry an explicit +Inf bucket, this drops the last real upper bound and never adds the implicit overflow bucket. That is the normal shape for Prometheus-derived Vector histograms—the parser removes the +Inf bucket (src/sources/prometheus/parser.rs:134-138) and the Prometheus exporter reconstructs it from count (src/sinks/prometheus/collector.rs:133-159). Encoding such a metric as OTLP turns a bucket like le=2 into the implicit +Inf bucket, and if there were observations above the last finite bound the OTLP count no longer matches the sum of bucket_counts; keep all finite bounds and append count - sum(bucket_counts) only when there is no infinity bucket.
Useful? React with 👍 / 👎.
| let (resource, scope, attributes) = split_metric_tags(tags); | ||
|
|
||
| let kind = metric.kind(); | ||
| let name = metric.name().to_string(); |
There was a problem hiding this comment.
When a native Vector metric has a namespace, this emits only Metric::name(), but the namespace is part of the metric series and is included by the existing metric sinks (for example Prometheus uses encode_namespace(metric.namespace().or(default_namespace), '_', metric.name()) in src/sinks/prometheus/collector.rs:41). Exporting namespaced metrics through the new OTLP serializer therefore renames/collides series such as vector.requests and app.requests into the same OTLP metric name requests; include the namespace somewhere in the encoded OTLP identity instead of discarding it.
Useful? React with 👍 / 👎.
| match tag_set { | ||
| TagValueSet::Empty => None, | ||
| TagValueSet::Single(tag) => Some(tag.clone()), | ||
| TagValueSet::Set(set) => set.iter().last().cloned(), |
There was a problem hiding this comment.
When a metric tag contains multiple values, this keeps only the last one, so native metrics whose tags were created through Vector's multi-value tag support lose label data on OTLP export. This is reachable from VRL (metric.set_multi_value_tag in lib/vector-core/src/event/vrl_target.rs:268) and existing metric encoders such as Datadog preserve every value via iter_all() (src/sinks/datadog/metrics/encoder.rs:724-730); if OTLP cannot represent these as repeated keys, encode an array value or return an error rather than silently dropping values.
Useful? React with 👍 / 👎.
| let timestamp_ns = metric | ||
| .timestamp() | ||
| .and_then(|ts| ts.timestamp_nanos_opt()) | ||
| .unwrap_or_else(|| Utc::now().timestamp_nanos_opt().unwrap_or(0)) | ||
| as u64; |
There was a problem hiding this comment.
For native metrics with a timestamp before 1970-01-01T00:00:00Z, timestamp_nanos_opt() returns a negative i64 and this cast wraps it into a huge u64, so the OTLP point is emitted with a far-future timestamp instead of the original time or an error. Vector metrics can carry user-provided timestamps, so pre-epoch or otherwise invalid OTLP timestamps should be rejected or clamped before converting to the unsigned OTLP field.
Useful? React with 👍 / 👎.
| MetricValue::Counter { value } => Ok(Data::Sum(Sum { | ||
| data_points: vec
Summary
Vector configuration
How did you test this PR?
Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References
Notes
@vectordotdev/vectorto reach out to us regarding this PR.pre-pushhook, please see this template.make fmtmake check-clippy(if there are failures it's possible some of them can be fixed withmake clippy-fix)make testgit merge origin masterandgit push.Cargo.lock), pleaserun
make build-licensesto regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.