feat(beholder): reduce PerWorkflow histogram buckets 16→8#2226
Draft
pkcll wants to merge 6 commits into
Draft
Conversation
AsCmdEnv always emitted CL_TELEMETRY_METRIC_CARDINALITY_LIMIT, so callers that don't set the field explicitly forced 0 and disabled the 10000 default in EnvConfig.parse. Only emit the env var when non-zero. mergeMetricViews prepended the default cardinality-limiting views (including a catch-all "*" view) before caller-supplied MetricViews. Since the OTel SDK dedupes views by stream identity and keeps the first match, this silently shadowed caller customizations such as histogram buckets set via WithOtelViews. Append defaults after caller views instead.
Correct the metricviews package/DefaultViews and mergeMetricViews comments to describe the actual OTel SDK behavior: an instrument may match multiple views, and dedup is by resolved stream identity (name/description/unit/kind) keeping the first in registration order — not "first matching view for the instrument". Explains why caller views precede defaults and why attribute filters do not compose. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
📊 API Diff Results
|
Add DefaultViews aggregation overrides for *.PerWorkflow.* histograms emitted by pkg/settings/limits. Uses 7 explicit boundaries (8 Prometheus buckets) per unit class: bytes (By), seconds (s), and dimensionless counts. CRE Limits-Workflow dashboards query histogram_quantile(1, ...) on these metrics, so coarser buckets preserve max usage vs limit panels while halving le cardinality on the dominant series families. Co-authored-by: Cursor <cursoragent@cursor.com>
pkcll
force-pushed
the
beholder-perworkflow-histogram-buckets
branch
from
2c01364 to
1f04f83
Compare
… views
The bucket-boundary views for *.PerWorkflow.* histograms claimed the OTel
stream identity ahead of the global deny-filter view, so the SDK's
first-match-wins dedup silently dropped the deny filter for those metrics.
Attach globalHighCardinalityDeny directly to each bucket-override Stream
mask so it travels with the aggregation override.
Also add a {gas}-unit-specific view: PerWorkflow gas limit histograms
(ChainWrite defaults up to 50_000_000) were falling into the unit-less
count-boundaries view (max 1e5), collapsing nearly all observations into
the +Inf bucket.
pkcll
force-pushed
the
beholder-otel-limit-metric-cardinality
branch
from
1203e1b to
8231c54
Compare
bolekk
reviewed
Jul 9, 2026
Contributor
There was a problem hiding this comment.
what happens if this limit is exceeded?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #2225. Reduces CRE PerWorkflow limit histograms from the OTel SDK default (15 boundaries / 16 buckets) to 7 boundaries / 8 buckets via Beholder
metricviews.DefaultViews().Targets
*.PerWorkflow.*histograms frompkg/settings/limits(bound,time,resource,range,queue,rate). Chainlink does not register caller views for these names, so the Beholder defaults apply.By(bytes)s(seconds)Motivation
Cardinality report for
linkforest-cre-don-workflow(~226k series): PerWorkflow + platform_engine histogram families dominate. Halving bucket count on PerWorkflow families cuts ~60k series on that node profile (590 workflows × ~30 histogram metrics × 8 vs 16 buckets).Alert impact
~/repos/alerts— no rules reference*_PerWorkflow_*_bucketorhistogram_quantileon PerWorkflow histograms.Dashboard impact
Limits-Workflow(13 zones)histogram_quantile(1, …)on 44 PerWorkflow*_bucketmetrics (max usage / usage-vs-limit %). No p95/p99.Workflows-Executions(13 zones)quantile(1)pattern in limit overview panel.gate_*/queue_*/resource_*PerWorkflow panelsWorkflows-Engine/platform_engine_*Staging validation: spot-check Limits-Workflow usage-vs-limit panels, especially byte limits near 100 MiB (top bucket boundary
1e8vs limit104,857,600B).Merge order
Merge #2225 first, then this PR.
Test plan
go test ./pkg/beholder/metricviews/...count({csa_public_key="…"})on a high-workflow-count node before/after