feat: add native continuous batching benchmark

[DilanRG]

Note

This remains a draft until it has had a loaded-model smoke test.

Summary

Adds a continuous-batching benchmark whose execution core is native C++. Python is limited to CLI validation, normal generation_inputs marshalling through ctypes, synthetic prompt construction, and console formatting.

The native runner:

• submits the whole cohort atomically into KoboldCpp's existing continuous-batching scheduler;
• uses the existing slot assignment and waiting queue, so jobs above --parallelrequests exercise real overflow waves;
• owns benchmark exclusivity against normal batch and legacy generation, and clears residual legacy sequence-0 KV before measuring;
• waits and aggregates in C++, including queue-inclusive cohort latency and throughput;
• rolls back partial submission on allocation/start failure and fails the full cohort on native decode errors;
• validates identical exact-size prompts, exact completion counts, and complete finite timing data before reporting PASS.

There is no second scheduler or parallel Python request runner. The existing native BatchRequest lines from 20e5b61 remain the source of per-request PP/TG timings.

Output

The console reports:

• jobs, native slots, overflow jobs, and minimum waves;
• engine and per-job context, prompt-token range, and total prompt/generated tokens;
• cohort end-to-end generated tokens/s, requests/s, and jobs/hour;
• queue-inclusive end-to-end latency p50/p95/max;
• native status/failure details and a curated non-secret snapshot of requested launch settings.

Metric-scope lines state that BatchRequest PP/TG excludes native queue time, cohort latency includes it, and HTTP/SSE/Horde/network overhead is outside this benchmark.

CLI

koboldcpp.exe model.gguf --benchmark --benchmark-batched 8 \
  --parallelrequests 4 --contextsize 40960 --benchmark-context 10240 \
  --genlimit 300

With four slots and eight jobs, this runs two native queue waves. A bare --benchmark-batched defaults jobs to the slot count. Omitting --benchmark-context divides loaded context across the jobs that can be active concurrently.

The synthetic benchmark alone uses deterministic greedy sampling and bans EOS. It does not normalize or override sampler settings for API or Horde generations.

Legacy --benchmark with --parallelrequests > 1 is rejected with guidance to add --benchmark-batched: the legacy path launches only one request and reads legacy timing globals, so its result is not a valid concurrent-batching measurement.

Validation

• 11 focused Python tests pass, covering the 112-byte native ABI, private one-call dispatch, workload/overflow sizing, exact prompt construction, reporting, failure output, and requested-config redaction.
• C++17 syntax checks pass for the modified native translation units in both CPU and CUDA configurations.
• C++ standard-layout, trivially-copyable, size, and key-offset assertions mirror the ctypes checks.
• git diff --check and CLI help checks pass.
• Independent concurrency and maintainability re-audits found no remaining blocker.

A loaded-model runtime smoke is still outstanding, so the PR remains a draft.

[DilanRG]

Quick status note: I'm reworking this draft into a much smaller implementation that reuses the existing continuous-batching scheduler and the new per-request timing output from 20e5b61. The plan is for Python to launch a synchronized cohort of normal generate() calls, with only an aggregate cohort/overflow summary added on top—no second native benchmark scheduler or timing model. Please don't spend time adapting the current draft; I'll force-update it once the simpler version is tested.

[DilanRG]

Update/correction to my earlier status note: the force-updated branch now uses a lean native C++ cohort runner, not a Python cohort of generate() calls. Python is only the CLI/ctypes/reporting bridge; atomic submission, real slot/overflow scheduling, waiting, validation, cleanup, and aggregate timing are all native and reuse the existing scheduler plus the BatchRequest PP/TG lines from 20e5b61. The focused tests and CPU/CUDA syntax checks pass. I am leaving the PR as a draft only until a loaded-model smoke test is available.