bdev/nvme: MPDBG debug prints for multipath failover diagnosis

[schmidt-scaled]

bdev/nvme: MPDBG debug prints for multipath failover diagnosis

Diagnostic instrumentation to resolve why, in the k8s_native_scale_break-20260716 incident, an ABORTED - SQ DELETION IO on the tertiary's LVS_31/hublvol multipath bdev was failed up to the hublvol layer instead of rerouted to the surviving path.

Why

Elimination against the stock code (bdev_nvme_io_complete_nvme_status → check_retry_io → any_io_path_may_become_available) left a contradiction: with bdev_retry_count=2, reconnect_delay=1, disable_read_ana_log_page=true, allow_accel_sequence=false, and no bdev-level reset, the aborted IO should have been requeued/rerouted — yet the log shows a synchronous fail-up. The NOTICE-level logs don't expose the any_io_path / find_io_path decision, so we can't see whether the second path was actually a live io_path at the failing instant. These prints expose exactly that.

What (strictly additive — no logic changes)

All prints are MPDBG-prefixed and fire only on the error/retry path:

• bdev_nvme_io_complete_nvme_status — sct/sc/dnr, retry_count vs bdev_retry_count, accel_seq, and the exit taken: GATE1 (dnr/aborted/retry-exhausted), GATE2 (accel_sequence), check_retry_io=TRUE→requeue, or check_retry_io=FALSE→fail-up.
• bdev_nvme_check_retry_io — the four branch predicates (path_err/sq_del/io_avail/ctrlr_avail) and the reroute-vs-fail-up decision.
• any_io_path_may_become_available — one line per io_path (ctrlr, ana_state, ana_timedout, ns_updating, qpair_connected, ctrlr_failed) + verdict. This is the crux.
• bdev_nvme_retry_io — whether a queued retry re-dispatches on the same io_path or re-selects.

The diff removes zero existing statements; every if/condition/control-flow line is byte-identical to the base branch. This is not intended to merge — it's a temporary diagnostic build.

Use

Build + deploy, reproduce the tertiary outage for LVS_31, then:

grep MPDBG spdk-container_*.log | grep -A20 'sq_del=1'

The any_io_path per-path dump at the abort instant decides it: either both hub paths were unavailable (stock fail-up is correct) or a live second io_path was skipped (real bdev_nvme bug).

🤖 Generated with Claude Code