Skip to content

Vertex AI default environment has broken cupy installation #709

Description

@ncclementi

In the altest Vertex AI env, the cuda toolkit version is 13

ls -ld /usr/local/cuda*
lrwxrwxrwx  1 root root   21 Jun  1 04:54 /usr/local/cuda -> /usr/local/cuda-13.0/
drwxr-xr-x 17 root root 4096 Jun  1 04:55 /usr/local/cuda-13.0

but when checking pip list they have a cuda 12 installation

cupy-cuda12x                          13.6.0

Which is a broken env:

import cupy as cp
X = cp.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1],[3, 2]], cp.float32)
X**2
RuntimeError: CuPy failed to load libnvrtc.so.12: OSError: libnvrtc.so.12: cannot open shared object file: No such file or directory

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions