bdev/nvme: fail I/O over to another path on generic errors

[schmidt-scaled]

bdev/nvme: fail I/O over to another path on generic errors

Problem

In a multipath setup where each path leads to an independent controller (primary / secondary / tertiary over a distributed backend), an I/O can complete with a non-path (generic) NVMe error on a path whose qpair and controller still look healthy — the completion arrives before the qpair is torn down, so it is not classified as a path error.

bdev_nvme_check_retry_io() handled this in its else branch by retrying on the same path and charging the attempt against the shared bdev_retry_count budget (default 3).

The reroute decision hinges entirely on spdk_nvme_cpl_is_path_error(), which is just status.sct == SPDK_NVME_SCT_PATH. A path that is dying but reports a generic status (and whose qpair/ctrlr liveness flags have not flipped yet) never trips that check, so:

• the same-path retries burn the very budget the cross-path failover also needs — both bdev_nvme_io_complete_nvme_status() (give-up) and the -ENXIO reroute in bdev_nvme_io_complete() gate on retry_count < bdev_retry_count;
• with a fast-failing dead path, retry_count reaches bdev_retry_count before the controller reset marks the qpair down;
• the I/O is failed upward even though a healthy alternate path existed the whole time.

Observed in a tertiary (lvs_31) hub-lvol topology: one I/O landed on the primary connection to a node that had just gone offline, completed in failure before the qpair was destroyed, triggered a controller reset, and was aborted upward instead of being retried on the healthy secondary.

Fix

Add a bounded, path-aware failover branch to bdev_nvme_check_retry_io():

1. Reroute instead of same-path retry. On a retryable non-path error, if a different io_path is available, fail the I/O over to it.
2. Separate accounting. Failovers are counted against a new per-I/O path_failover_count, not retry_count, so dead-end attempts no longer consume the same-path retry budget.
3. Skip the failed path. io_path_to_avoid is honored by _bdev_nvme_find_io_path() for exactly one (synchronous) selection — otherwise active-passive would just re-pick the same still-"available" path and never reach the alternate. If the avoided path turns out to be the only usable one, it falls back to it rather than failing the I/O.
4. Termination guaranteed. The number of failovers is bounded by the number of io_paths, and we only fail over while another path is available. A genuinely deterministic error (e.g. a media error returned identically by every path) still terminates: once every path has been tried the I/O falls through to the existing same-path retry branch and is reported upward as before.

Safety / scope

• Single-path configs are unaffected — with no other available path the failover branch is skipped; behavior is byte-for-byte identical to before. Also covers DNR / aborted-by-request / accel-sequence (still filtered by the caller) and hard path errors (unchanged branch).
• The min_qd (queue-depth) selector is intentionally left unchanged; it already balances by outstanding-request count and the reported issue is specific to active-passive / round-robin.
• io_path_to_avoid on the channel is a transient hint, set and cleared around a single synchronous find_io_path() call in bdev_nvme_submit_request(), so it can never leak into another I/O's selection.

Files

• module/bdev/nvme/bdev_nvme.h — transient io_path_to_avoid on nvme_bdev_channel.
• module/bdev/nvme/bdev_nvme.c — path_failover_count + io_path_to_avoid on nvme_bdev_io; bdev_nvme_num_io_paths() / bdev_nvme_another_io_path_is_available() helpers; skip logic in _bdev_nvme_find_io_path(); failover branch in bdev_nvme_check_retry_io(); hint plumbing in bdev_nvme_submit_request(); field resets on completion.

Testing

• Build (make in a Linux SPDK tree) — not run in the authoring environment.
• test/unit/lib/bdev/nvme/bdev_nvme.c/bdev_nvme_ut — the existing "generic error retries on the same path with two paths" expectations will change and need updating to assert failover + counter separation.
• Reproduce the incident: two paths to independent controllers, active-passive, take the active path's node offline mid-I/O, confirm outstanding I/O fails over to the peer instead of being aborted upward, and that bdev_retry_count is not exhausted by the dead path.