Memory Copies from Device Significantly Slower than CUDA

[zfergus]

Thank you for providing this library. I have found it immensely helpful when using Vulkan compute shaders.

I am profiling my application, and I found that copying back to the CPU takes significantly more time than in CUDA. Specifically, this line cost 95% of my run time:

std::vector<std::pair<uint32_t, uint32_t>> candidates = 
    t_candidates->vector<std::pair<uint32_t, uint32_t>>();

where t_candidates is created as

auto t_candidates = mgr->tensor(
    candidates_size, 2 * sizeof(uint32_t),
    kp::Memory::DataTypes::eCustom, kp::Memory::MemoryTypes::eDevice);

I tried to reproduce this same effect in an as-simple-as-possible example using the Python bindings for Komput and PyCUDA. I have attached a PDF of the notebook. You can see copying 51 MiB from the GPU to CPU costs 7.3 ms ± 399 µs for CUDA but 292 ms ± 160 µs for Kompute.

Is this an inherent limitation of Vulkan, or is there a way to speed up this copy?

---

Here are my GPU specs:

Device: NVIDIA GeForce RTX 5080
Compute Capability: (12, 0)
Total Memory: 15817 MiB
Max threads per block: 1024
Total number of SMs: 84

---

Attachments:
cuda_vs_kompute.pdf

[axsaucedo]

Thank you for opening the issue, what is the size of your tensor? Copy operations tend to be the most costly which is why it's set up to reduce the copy operations (e.g. doing them only once at the beginning), which means that operations would have to be optimized for either mostly CPU or GPU whilst minimising data transfers. Are you finding issues specifically on Kompute? I suspect that this is just a costly copy operation but let us know otherwise.

[zfergus]

I am copying a tensor of 13,472,684 containing uint32_t. I have not setup the same example in Vulkan directly, but have set it up in CUDA. The CUDA memory transfer is 40x faster, so it doesnt seem like its a hardware limitation.

[axsaucedo]

Strange - looking at the code I can't find any step that could result in taking too long on the CPU side, namely the call path is:

Manager::tensor ->
Tensor::Tensor ->
Memory::Memory ->
Tensor::reserve ->
Tensor::allocateMemoryCreateGPUResources

From what I see there doesn't seem to be any CPU intensive steps in any of the call stack steps, and the only intensive steps are GPU-specific in allocateMemoryCreateGPUResources which are minimal vulkan logic to create the buffer.

The only consideration that we can take is that here we do create two buffers, the primary (GPU-only) and the secondary (CPU-visible for transfer) buffer - potentially the CUDA one may only create the former but even then I would only see a 2x speedup here (unless the former is the one that takes significantly longer).

If you step through your code, is there one specific line that takes most of that time? Otherwise we could assume that indeed given it's an NVIDIA GPU then CUDA would be extremely highly optimized for data transfers vs Vulkan not being as optimized, but 40x to me sounds surprising.

Would be great if we can get more granular view of this performance - is that something you can share?

[zfergus]

Yes, I'll try share a more granular timing. To clarify it is the copy from the GPU to the CPU that is time consuming. The CPU to GPU copy is fast.

[zfergus]

Here is a more detailed breakdown of my original example, including timings for key blocks.

void detect_candidates(
    std::shared_ptr<kp::Manager> mgr,
    std::shared_ptr<kp::Tensor> t_lbvh_source,
    std::shared_ptr<kp::Tensor> t_lbvh_target,
    const std::function<bool(index_t, index_t)>& can_collide,
    uint32_t predicted_candidates_size,
    std::vector<std::pair<int, int>>& candidates) const
{
    assert((t_lbvh_source->size() + 1) % 2 == 0);
    const uint32_t num_elements_source = (t_lbvh_source->size() + 1) / 2;

    auto t_collision_count =
        mgr->tensorT<uint32_t>(std::vector<uint32_t> { 0 });

    auto t_candidates = mgr->tensor(
        predicted_candidates_size, 2 * sizeof(uint32_t),
        kp::Memory::DataTypes::eCustom, kp::Memory::MemoryTypes::eDevice);

    std::vector<std::shared_ptr<kp::Memory>> params = { {
        t_lbvh_source,
        t_lbvh_target,
        t_collision_count,
        t_candidates,
    } };

    uint32_t triangular = 0;
    auto algorithm = mgr->algorithm(
        params, shader_detect_candidates,
        kp::Workgroup({ { blocks_per_grid(num_elements_source), 1, 1 } }),
        std::vector<uint32_t> { triangular },
        std::vector<uint32_t> { num_elements_source });

    {
        PROFILE_BLOCK("shader");
        mgr->sequence()
            ->record<kp::OpSyncDevice>(params)
            ->record<kp::OpAlgoDispatch>(algorithm)
            ->eval();
    } // runtime: 8.351 ms

    {
        PROFILE_BLOCK("sync collision count");
        mgr->sequence()
            ->record<kp::OpSyncLocal>({ { t_collision_count } })
            ->eval();
    } // runtime: 0.753 ms

    const uint32_t collision_count = *(t_collision_count->data());

    if (collision_count == 0) {
        return;
    }
    assert(collision_count <= predicted_candidates_size);

    {
        PROFILE_BLOCK("sync candidates");
        mgr->sequence()->record<kp::OpSyncLocal>({ { t_candidates } })->eval();
    } // runtime: 2.902 ms

    std::vector<std::pair<uint32_t, uint32_t>> candidates_ui32;
    {
        PROFILE_BLOCK("t_candidates->vector");
        candidates_ui32 = t_candidates->vector<std::pair<uint32_t, uint32_t>>();
    } // runtime: 690.463 ms

    {
        PROFILE_BLOCK("filter candidates");
        // Apply the can_vertices_collide filter
        candidates.reserve(candidates_ui32.size());
        for (size_t ci = 0; ci < candidates_ui32.size(); ++ci) {
            const auto& [i, j] = candidates_ui32[ci];
            assert(!triangular || i < j); // No duplicates
            if (!can_collide(i, j)) {
                continue;
            }
            candidates.emplace_back(i, j);
        }
        candidates.shrink_to_fit(); // Remove excess capacity
    } // runtime: 12.320 ms
} // runtime: 752.779 ms

The line candidates_ui32 = t_candidates->vector<std::pair<uint32_t, uint32_t>>(); takes up 92% of the total runtime.

[axsaucedo]

Ah very interesting, thank you for sharing - two observations: 1) it seems that most of the inefficiency may be primarily on the data structure used, and on this 2) it seems from online search that it may be std::pair has complaints of bad performance so I wonder if this may be the case.

My suggestion would be that you attempt the same code with an std::vector<uint32_t>where you handle he pairs through indexing instead of directly through anstd::pair`, and it may address the performance issues.

[zfergus]

Thank you for the suggestion. I'll try that out.

[axsaucedo]

Just following up to see if there was any insights from the exploration

[zfergus]

Thanks for following up. I tested the suggested changes, but unfortunately, there was no major speed-up by using std::vector<uint32_t> instead of std::vector<std::pair<uint32_t, uint32_t>>.

I can share the Jupyter notebook to see if you get the same performance. GitHub won't let me share the file directly.

[axsaucedo]

Hmm strange - yes I think if you can share the jupyter notebook we can have a deeper look

[zfergus]

See if you can access the notebook here.