refactor: add new module for backend-specific code

[wbruna]

This is an initial attempt to replace build-time backend checks with runtime ones.

My main intention is to eventually be able to decouple all code from fixed ggml backends, which would give us the ability to have a distributed binary for any CPU instruction set, use more than one GPU backend type at the same time, etc. But even if that's not achievable (or desired!), I believe Koboldcpp can still benefit from changes in that direction: reduced build times, lower chance of building OK on one backend and failing on another, smaller binaries, etc.

I'm submitting very early to validate the approach before spending too much time on this, so right now it only replaces some easier parts of gpttype_adapter: checks for tensor splitting, BLAS support, etc. I've also refactored the tensor splitting check to avoid a bit of repetitive code.

The runtime backend tests are very similar to the ones used by stable-diffusion.cpp, identifying the backends by their names (with a bit of convenience code to be able to check several at once). Right now, I'm always checking the first device, so it should give the same results as the compile-time checks; but the API is already prepared to accept backend pointers.

There should be only one functional change: the check

    #if defined(GGML_USE_CUDA)
    mtmd_fa = LLAMA_FLASH_ATTN_TYPE_DISABLED; //kcpp: disabled in 1.102.2 as some headsizes break on turing
    #endif

also triggered for ROCm; so I've replaced it with a straight CUDA-only check.

[LostRuins]

also triggered for ROCm; so I've replaced it with a straight CUDA-only check.

i don't know if the bug also affects rocm or not.

also, i think the metal device names might be wrong

and i think ggml_backend_dev_get(0); is not correct for multi-gpu systems.

[wbruna]

i don't know if the bug also affects rocm or not.

Well, I was following the comment. My own card doesn't seem to be affected.

Do you have a reference for the bug? The closest I've found (issue 16950) is just a warning, and doesn't fit the timeline for that release.

also, i think the metal device names might be wrong

Found it: ggml/src/ggml-metal/ggml-metal-device.m places "MTL%d" in the name field. I'll fix it.

and i think ggml_backend_dev_get(0); is not correct for multi-gpu systems.

It is correct at least on Vulkan. But I could add a test for skipping non-GPUs/iGPUs, just to be sure.

[LostRuins]

It is correct at least on Vulkan. But I could add a test for skipping non-GPUs/iGPUs, just to be sure.

what i mean is, its possible for other backends to be initialized first e.g. BLAS, so index 0 is not the only index to check

[wbruna]

what i mean is, its possible for other backends to be initialized first e.g. BLAS, so index 0 is not the only index to check

But could that happen with current Koboldcpp? Both CUDA0 and BLAS0 appearing on the device list on the same run? (I'm not familiar with the BLAS backend).

[LostRuins]

Nope, because current koboldcpp defines are set at compile time for each dll, so if you pick that library, it will be set regardless of what devices or backends you have configured on your system.

[wbruna]

Nope, because current koboldcpp defines are set at compile time for each dll, so if you pick that library, it will be set regardless of what devices or backends you have configured on your system.

Should be safe then.

These tests are not adequate for a future, possibly-multi-backend version. But right now the tests are assuming a single type of GPU, to make the transition easier to do piece by piece: just replacing the global ifdefs with the 'global' GPU type. The backend type will still be dictated by which ggml runtime is used at link time.

What each test should be doing is passing the ggml_backend_t pointer, to test the backend directly. But that involves a bigger diff, to store and propagate the backend pointers, so I'd rather leave it to a second phase.

[wbruna]

I believe the latest changes are enough to avoid per-backend builds of most code files (excluding ggml itself, of course).

One question, though: why do we have per-CPU-type gpttype_adapter.cpp builds? I'd expect it to matter for ggml, but...

[LostRuins]

One question, though: why do we have per-CPU-type gpttype_adapter.cpp builds? I'd expect it to matter for ggml

there used to be CPU intrinsic related codes there, i think they've been refactored out already though

[wbruna]

I've changed the build to reuse gpttype_adapter.o and whispercpp_default.o objects on all builds. Seems to be working, both on Vulkan and ROCm!