Set trace sampling as the default behavior #8554
Description
Description
We received a report about high resource consumption with OpenTelemetry enabled. The user also enabled alloy traces + profiling which may have further harmed performance. Regardless of their specific set-up, we should default to trace sampling as it is effective at reducing resource consumption without losing out on observability
https://opentelemetry.io/docs/concepts/sampling/#why-sampling
The default tracing value should be around 1%, and should be configurable via a documented CLI flag
The current work around
Using these env variables enables trace sampling at 1%
OTEL_TRACES_SAMPLER=parentbased_traceidratio,OTEL_TRACES_SAMPLER_ARG=0.01
Requiring users to supply these env variables is bad UX, especially since these are not documented within the Lighthouse book. Most users will probably want 1% trace sampling by default
Error report
we got a lot of errors and warnings about high load and missed many blocks in our ssv node cluster. It was not 100% clear if it was only because of the setting up of alloy with traces + profiles (my understanding is that it even takes more resources than traces). And of course we run a lot of different client combinations so it was most likely a combination of different problems.
2025-12-04 17:34:50.708
error
Dec 04 17:34:50.707 ERROR Block broadcast was delayed msg: "system may be overloaded, block may be orphaned", provenance: "builder", delay_ms: 3707, slot: 13170172, root: 0xa517287f765593a6bdde09c53d317540404547319419eeb9ecfd57517c2daf62
2025-12-04 15:42:39.330
error
Dec 04 15:42:39.330 ERROR Block was broadcast too late msg: "system may be overloaded, block likely to be orphaned", provenance: "builder", delay_ms: 4330, slot: 13169611, root: 0x0f9dfbbd83f3a864358cde181df9a6bcce0691dc9162f74c5a9901cab007fe59
2025-12-04 14:09:24.664
error
Dec 04 14:09:24.664 WARN Invalid execution payload validation_error: Some("expected blob versioned hashes do not match the given transactions"), latest_valid_hash: None, execution_block_hash: 0xdc3d3d9df7052d07b86ed9813d079cbf5941541ef4b843cc7cfa948f9305b863, root: 0x4958529b043ec60f34806769f0d3b0e5327bd43804e9343f4700fc6c30a2c6d2, graffiti: "", proposer_index: 1350051, slot: 13169145, method: "new_payload"