Cycles - Rendering on a Nvidia A100 crashes/fails on Google Colab #91879

System Information
Operating system:

NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"

Graphics card: Nvidia A100-SXM4-40GB

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      Off  | 00000000:00:04.0 Off |                    0 |
| N/A   34C    P0    42W / 400W |      0MiB / 40536MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

Blender Version
Blender 2.93.0-2.93.4, Blender 3.0.0
Worked: Blender 2.92.0

Short description of error
Blender is crashing and aborting the render with Cycles

Exact steps for others to reproduce the error
Open Google Colab
Try to get a A100
Run https://colab.research.google.com/gist/Raimund58/88061f785ea2563d345914d196c15b87/blender_script_for_google_colab_using_the_gpu.ipynb
It will crash

blender.crash_2.93.4.txt
Crashlog_2.93.4.txt
Crashlog_3.0.0.txt

CC @brecht @pmoursnv

**System Information** Operating system: NAME="Ubuntu" VERSION="18.04.5 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.5 LTS" VERSION_ID="18.04" Graphics card: Nvidia A100-SXM4-40GB ``` +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.63.01 Driver Version: 460.32.03 CUDA Version: 11.2 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 A100-SXM4-40GB Off | 00000000:00:04.0 Off | 0 | | N/A 34C P0 42W / 400W | 0MiB / 40536MiB | 0% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ ``` **Blender Version** Blender 2.93.0-2.93.4, Blender 3.0.0 Worked: Blender 2.92.0 **Short description of error** Blender is crashing and aborting the render with Cycles **Exact steps for others to reproduce the error** Open Google Colab Try to get a A100 Run https://colab.research.google.com/gist/Raimund58/88061f785ea2563d345914d196c15b87/blender_script_for_google_colab_using_the_gpu.ipynb It will crash [blender.crash_2.93.4.txt](https://archive.blender.org/developer/F10713054/blender.crash_2.93.4.txt) [Crashlog_2.93.4.txt](https://archive.blender.org/developer/F10713060/Crashlog_2.93.4.txt) [Crashlog_3.0.0.txt](https://archive.blender.org/developer/F10713050/Crashlog_3.0.0.txt) CC @brecht @pmoursnv

Added subscribers: @pmoursnv, @brecht, @Raimund58

Since Blender does not ship with pre-compiled kernels for A100, it falls back to the PTX one it ships with. But the PTX files that ship with Blender 2.93.4 and 3.0.0 were compiled with CUDA 11.4 (PTX version 7.4).
This requires a r470 driver or newer, older drivers are not able to interpret that PTX version and will fail to compile the kernels. You are running r460, which only supports up to PTX version 7.2, and therefore it fails (and Blender does not handle the error correctly it seems and just crashes shortly after).

Since Blender does not ship with pre-compiled kernels for A100, it falls back to the PTX one it ships with. But the PTX files that ship with Blender 2.93.4 and 3.0.0 were compiled with CUDA 11.4 (PTX version 7.4). This requires a r470 driver or newer, older drivers are not able to interpret that PTX version and will fail to compile the kernels. You are running r460, which only supports up to PTX version 7.2, and therefore it fails (and Blender does not handle the error correctly it seems and just crashes shortly after).

Changed status from 'Needs Triage' to: 'Needs User Info'

Ah, I was not aware that upgrading the CUDA toolkit would affect driver compatibility this way. Not sure it's worth reverting to an older CUDA version for this case, I think mentioning it in the 2.93 release notes is enough.

@pmoursnv, can we improve the test here to give a better error message, and then backport that to 2.93?

@Raimund58, can you verify if upgrading the driver to 470 fixes the problem?

Ah, I was not aware that upgrading the CUDA toolkit would affect driver compatibility this way. Not sure it's worth reverting to an older CUDA version for this case, I think mentioning it in the 2.93 release notes is enough. @pmoursnv, can we improve the test here to give a better error message, and then backport that to 2.93? @Raimund58, can you verify if upgrading the driver to 470 fixes the problem?

Sure, I can post a fix later today.
The driver reports CUDA_ERROR_UNSUPPORTED_PTX_VERSION (222), which is a more recent addition (was added in CUDA 11.1 as far as I can tell) and not currently in CUEW, hence the "Unknown CUDA error value" in the log.
The crash is then happening because Cycles tries to call "cuOccupancyMaxPotentialBlockSize" with a NULL kernel function in "CUDADevice::load_functions" (since the module failed to load).

Sure, I can post a fix later today. The driver reports `CUDA_ERROR_UNSUPPORTED_PTX_VERSION` (222), which is a more recent addition (was added in CUDA 11.1 as far as I can tell) and not currently in CUEW, hence the "Unknown CUDA error value" in the log. The crash is then happening because Cycles tries to call "cuOccupancyMaxPotentialBlockSize" with a NULL kernel function in "CUDADevice::load_functions" (since the module failed to load).

I am quite confused and I don't know if it is because Colab or if I am doing something wrong.
The system said that 470 is already installed, no? But why is nothing changing?

First try to update the driver:

!cat /etc/os-release
!nvidia-smi
!sudo apt install nvidia-driver-470
!nvidia-smi

Log1.txt
(Driver already up to date)

Second try to update the driver:

I followed this site: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=18.04&target_type=deb_local

!cat /etc/os-release
!nvidia-smi
!wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
!sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
!wget https://developer.download.nvidia.com/compute/cuda/11.4.2/local_installers/cuda-repo-ubuntu1804-11-4-local_11.4.2-470.57.02-1_amd64.deb
!sudo dpkg -i cuda-repo-ubuntu1804-11-4-local_11.4.2-470.57.02-1_amd64.deb
!sudo apt-key add /var/cuda-repo-ubuntu1804-11-4-local/7fa2af80.pub
!sudo apt-get update
!sudo apt-get -y install cuda
!nvidia-smi

Log2.txt
(Still nothing changed)

What am I not seeing?

I am quite confused and I don't know if it is because Colab or if I am doing something wrong. The system said that 470 is already installed, no? But why is nothing changing? First try to update the driver: ``` !cat /etc/os-release !nvidia-smi !sudo apt install nvidia-driver-470 !nvidia-smi ``` [Log1.txt](https://archive.blender.org/developer/F10763062/Log1.txt) (Driver already up to date) Second try to update the driver: I followed this site: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=18.04&target_type=deb_local ``` !cat /etc/os-release !nvidia-smi !wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin !sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600 !wget https://developer.download.nvidia.com/compute/cuda/11.4.2/local_installers/cuda-repo-ubuntu1804-11-4-local_11.4.2-470.57.02-1_amd64.deb !sudo dpkg -i cuda-repo-ubuntu1804-11-4-local_11.4.2-470.57.02-1_amd64.deb !sudo apt-key add /var/cuda-repo-ubuntu1804-11-4-local/7fa2af80.pub !sudo apt-get update !sudo apt-get -y install cuda !nvidia-smi ``` [Log2.txt](https://archive.blender.org/developer/F10763074/Log2.txt) (Still nothing changed) What am I not seeing?

I'm not really familiar enough with Linux driver installation to be much help with that. Something is weird on the system though: The called nvidia-smi tool was installed by a r470 driver, but the actual driver running (as reported by nvidia-smi) is a r460 one ... As if both drivers are installed in parallel, but the r460 one is actually active. Wonder if one can force uninstall the old driver and then try to cleanly install r470 over it again.

Fix for the crash in 2.93 is here: https://developer.blender.org/P2478
Super simple, but I'm not sure about the process for backporting/committing stuff to 2.93 (or 2.83 for that matter, which suffers from the same bug)?

I'm not really familiar enough with Linux driver installation to be much help with that. Something is weird on the system though: The called nvidia-smi tool was installed by a r470 driver, but the actual driver running (as reported by nvidia-smi) is a r460 one ... As if both drivers are installed in parallel, but the r460 one is actually active. Wonder if one can force uninstall the old driver and then try to cleanly install r470 over it again. Fix for the crash in 2.93 is here: https://developer.blender.org/P2478 Super simple, but I'm not sure about the process for backporting/committing stuff to 2.93 (or 2.83 for that matter, which suffers from the same bug)?

@Raimund58.: istalling the CUDA toolkit should not be needed, just the driver. Not sure what is going on with that driver though. Maybe needs a system restart or kernel module reload. If it takes a lot of time to investigate that it's probably not worth it just to confirm if this bug is fixed.

@pmoursnv: you can add it to the list in #88449 (Blender LTS: Maintenance Task 2.93) and #77348 (Blender LTS: Maintenance Task 2.83). Linking to the patch from there is fine, Jeroen should be able to commit that..

@Raimund58.: istalling the CUDA toolkit should not be needed, just the driver. Not sure what is going on with that driver though. Maybe needs a system restart or kernel module reload. If it takes a lot of time to investigate that it's probably not worth it just to confirm if this bug is fixed. @pmoursnv: you can add it to the list in #88449 (Blender LTS: Maintenance Task 2.93) and #77348 (Blender LTS: Maintenance Task 2.83). Linking to the patch from there is fine, Jeroen should be able to commit that..

@brecht The Google Colab session can not rebooted afaik. So, no fix for me :/
Guess I have to wait for Google to update the driver.
I will come back in case I find a workaround.

@brecht The Google Colab session can not rebooted afaik. So, no fix for me :/ Guess I have to wait for Google to update the driver. I will come back in case I find a workaround.

This issue was referenced by c11585a82f

This issue was referenced by c11585a82f97e51c01c4f4f309b85bdf7602ca08

@Raimund58 You could build Blender yourself with CUDA toolkit 11.2, that should give you PTX files that are compatible. Or build a Cubin for sm_80 directly (via CYCLES_CUDA_BINARIES_ARCH CMake variable), then the CUDA toolkit version should not matter.
Blender doesn't ship with a Cubin for sm_80 by default since there are no consumer GPUs using that architecture.

@brecht I don't have permission to edit those lists, could you add this?:

Report	Commits in Master
#91879	`c11585a82f`	Use P2478 instead (has additional crash fix)

@Raimund58 You could build Blender yourself with CUDA toolkit 11.2, that should give you PTX files that are compatible. Or build a Cubin for sm_80 directly (via `CYCLES_CUDA_BINARIES_ARCH` CMake variable), then the CUDA toolkit version should not matter. Blender doesn't ship with a Cubin for sm_80 by default since there are no consumer GPUs using that architecture. @brecht I don't have permission to edit those lists, could you add this?: | Report | Commits in Master | | | -- | -- | -- | | #91879 | c11585a82f | Use [P2478](https://archive.blender.org/developer/P2478.txt) instead (has additional crash fix) |

Added subscriber: @LazyDodo

@pmoursnv added you to the right groups, permissions should be ok now.

Thanks! Done.

Changed status from 'Needs User Info' to: 'Resolved'

I'll consider this resolved then.

This issue was referenced by b382632665

This issue was referenced by b382632665b3552d580a3c65e94dd36857d5fb68

This issue was referenced by 31dfdb6379

This issue was referenced by 31dfdb6379cd42b919ba529eb9cfd3f29eb8de64

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Labels Milestones

Cycles - Rendering on a Nvidia A100 crashes/fails on Google Colab #91879