Add support for running sql benchmarks with command line arguments

[Omega359]

Which issue does this PR close?

• Part of [EPIC] Add SQL based benchmarking #21706 and CLI (non criterion) interface for sql runner #21937

Rationale for this change

SQL benchmark suites expose configuration through environment variables. Contributors must consult the source or documentation to discover suite-specific settings, and Cargo does not forward custom arguments to benchmark targets.

This change lets benchmark_runner expose those settings as validated command-line arguments. It also adds suite-specific help and a dry-run mode for inspecting resolved configuration without executing a benchmark.

What changes are included in this PR?

This PR adds TOML .suite metadata files that define each SQL benchmark suite’s:

• Description and help examples.
• Suite-specific options, defaults, environment variables, and accepted values.
• Query filename pattern.
• Data-path replacements.

benchmark_runner uses this metadata to:

• Generate suite-specific command-line arguments and help output.
• Resolve suite options using command-line, environment-variable, and default-value precedence.
• Support --path for suites that declare a DATA_DIR replacement.
• Support --result-mode for result persistence and validation.
• Validate suite names, metadata, option conflicts, query identifiers, and incompatible arguments.
• Provide --dry-run JSON output containing the resolved options, paths, value sources, filters, and execution mode without loading benchmark
  definitions or executing SQL.

The existing basic runner, Criterion mode, and suite-listing functionality remain in place.

Are these changes tested?

Yes. New tests cover:

• Suite metadata parsing and validation.
• Dynamic suite options and help output.
• Option precedence between command-line arguments, environment variables, and defaults.
• Path and query filename resolution.
• Dry-run output and validation.
• Result-mode resolution.
• Invalid option combinations and malformed metadata.

Are there any user-facing changes?

Yes. Contributors can configure suite-specific settings through command-line arguments and inspect them through suite help:

cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch --help

cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- \
  tpch --query 15 --format csv --path /path/to/tpch

cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- \
  clickbench --partitioning partitioned --dry-run

Environment variables remain supported for compatibility with direct cargo bench invocations and existing benchmark definitions.

The SQL benchmark README documents the command-line options, resolution precedence, dry-run behaviour, and .suite metadata format.

I have a script that cover a large number of combinations to exercise the benchmark_runner as much as possible.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 96.00232% with 69 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.77%. Comparing base (eef1017) to head (1c7a4be).

Files with missing lines	Patch %	Lines
benchmarks/src/bin/benchmark_runner.rs	95.28%	29 Missing and 17 partials ⚠️
benchmarks/src/sql_benchmark_suite.rs	96.74%	9 Missing and 10 partials ⚠️
benchmarks/src/sql_benchmark_runner.rs	97.60%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23772      +/-   ##
==========================================
+ Coverage   80.71%   80.77%   +0.06%     
==========================================
  Files        1089     1090       +1     
  Lines      368748   370359    +1611     
  Branches   368748   370359    +1611     
==========================================
+ Hits       297633   299171    +1538     
- Misses      53372    53416      +44     
- Partials    17743    17772      +29     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Omega359]

@alamb - this should be the last main PR for the cli. The remaining work is in the bench.sh script to use the benchmark_runner and updated documentation.