Blender crashing when trying to use Cupy #76414

Closed
opened 2020-05-04 19:35:58 +02:00 by Nathan Kundtz · 10 comments

System Information
Operating system: Linux-5.3.0-51-generic-x86_64-with-debian-buster-sid 64 Bits
Graphics card: GeForce GTX 1050/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 440.59
CUDA: 10.2

Blender Version
Broken: version: 2.83 (sub 13), branch: master, commit date: 2020-04-13 08:07, hash: blender/blender@2ec6eca518
Broken: version: 2.83 (sub 15), branch: master, commit date: 2020-05-03 18:42, hash: blender/blender@331bf04fad
Worked: version: 2.83 (sub 11), branch: master, commit date: 2020-03-31 23:07, hash: blender/blender@da3cb514e5

Short description of error
Blender crashes (Segmentation Fault) while trying to use cupy.
It will pass the first three flags and then segfault before the fourth.

CudaTest.py

Exact steps for others to reproduce the error
From the command line use blender --background --python CudaTest.py
This will throw a seg fault
If you run the same script outside of blender:
$(BLENDER_DIR)/2.83/python/bin/python3.7m CudaTest.py
The code will run normally.

Environment setup
Assuming CUDA is installed normally and from a fresh blender install:
$(BLENDER_DIR)/2.83/python/bin/python3.7m -m ensurepip
$(BLENDER_DIR)/2.83/python/bin/python3.7m -m pip install cupy-cuda102

Much appreciation
Really appreciate your help! Thanks and hope everyone is staying safe!

Other information
Example backtrace of the core dump below.

- 0  0x000000000d555b20 in nvrtcVersion ()
- 1  0x00007fca78ca297d in pyx_f_4cupy_4cuda_5nvrtc_getVersion (pyx_skip_dispatch=<optimized out>)
    at cupy/cuda/nvrtc.cpp:1923
#2  0x00007fca78ca2afb in pyx_pf_4cupy_4cuda_5nvrtc_2getVersion (pyx_self=<optimized out>)
    at cupy/cuda/nvrtc.cpp:2021
#3  pyx_pw_4cupy_4cuda_5nvrtc_3getVersion (pyx_self=<optimized out>, unused=<optimized out>)
    at cupy/cuda/nvrtc.cpp:2008
- 4  0x0000000005700a20 in _PyMethodDef_RawFastCallKeywords ()
- 5  0x0000000005700ab5 in _PyCFunction_FastCallKeywords ()
- 6  0x0000000000c0d098 in _PyEval_EvalFrameDefault ()
- 7  0x0000000000c04b80 in ?? ()
- 8  0x0000000000c0e8fe in _PyEval_EvalFrameDefault ()
- 9  0x0000000000c04b80 in ?? ()
- 10 0x0000000000c0e8fe in _PyEval_EvalFrameDefault ()
- 11 0x00000000057b244c in _PyEval_EvalCodeWithName ()
- 12 0x0000000005700536 in _PyFunction_FastCallKeywords ()
- 13 0x0000000000c0e8fe in _PyEval_EvalFrameDefault ()
- 14 0x00000000057b244c in _PyEval_EvalCodeWithName ()
- 15 0x0000000005700380 in _PyFunction_FastCallDict ()
#16 0x00007fcaa487fe57 in __Pyx_PyObject_Call (kw=0x7fca5234b0a0, arg=0x7fca523e7980, func=<optimized out>)
    at cupy/core/core.cpp:53653
#17 pyx_f_4cupy_4core_4core_compile_with_cache (pyx_v_source=0x7fca5247e1d0, 
    pyx_skip_dispatch=<optimized out>, pyx_optional_args=<optimized out>) at cupy/core/core.cpp:35823
#18 0x00007fca703887a1 in pyx_f_4cupy_4core_7_kernelget_simple_elementwise_kernel (
    pyx_v_params=0x7fca527131e0, pyx_v_arginfos=0x7fca5234b910, pyx_v_operation=0x7fca5233d2d0, 
    pyx_v_name=0x7fca5277d490, pyx_v_type_map=<optimized out>, pyx_v_preamble=0x7fcae7c0e3b0, 
    pyx_skip_dispatch=0, pyx_optional_args=0x7ffdf824d010) at cupy/core/_kernel.cpp:4481
#19 0x00007fca70389127 in pyx_pf_4cupy_4core_7_kernel_2_get_simple_elementwise_kernel (
    pyx_v_options=<optimized out>, pyx_v_after_loop=<optimized out>, pyx_v_loop_prep=0x7fcae7c0e3b0, 
    pyx_v_preamble=<optimized out>, pyx_v_type_map=<optimized out>, pyx_v_name=0x7fca5277d490, 
    pyx_v_operation=0x7fca5233d2d0, pyx_v_arginfos=0x7fca5234b910, pyx_v_params=0x7fca527131e0, 
    pyx_self=<optimized out>) at cupy/core/_kernel.cpp:4711
#20 pyx_pw_4cupy_4core_7_kernel_3_get_simple_elementwise_kernel (pyx_self=<optimized out>, 
    pyx_args=<optimized out>, pyx_kwds=<optimized out>) at cupy/core/_kernel.cpp:4681
- 21 0x0000000005702705 in PyCFunction_Call ()
- 22 0x00007fca703860ff in Pyx_PyObject_Call (kw=0x7fca52348500, arg=0x7fca523e78a0, func=0x7fca79748500)
    at cupy/core/_kernel.cpp:36022
#23 pyx_pf_4cupy_4core_7_kernel_12_get_elementwise_kernel (pyx_v_kwargs=0x7fca52348500, 
    pyx_v_preamble=0x7fcae7c0e3b0, pyx_v_name=0x7fca5277d490, pyx_v_operation=0x7fca5233d2d0, 
    pyx_v_params=0x7fca527131e0, pyx_v_type_map=0x7fcaa4af4750, pyx_v_arginfos=0x7fca5234b910, 
    pyx_self=<optimized out>) at cupy/core/_kernel.cpp:15617
#24 pyx_pw_4cupy_4core_7_kernel_13_get_elementwise_kernel (__pyx_self=<optimized out>,
pyx_args=<optimized out>, pyx_kwds=<optimized out>) at cupy/core/_kernel.cpp:15281
- 25 0x0000000005702705 in PyCFunction_Call ()
- 26 0x00007fca862468a7 in Pyx_PyObject_Call (kw=0x7fca52344eb0, arg=0x7fca523e77c0, func=0x7fca7975a410)
    at cupy/util.cpp:4384
#27 pyx_pf_4cupy_4util_7memoize_9decorator_ret (pyx_v_kwargs=0x7fca52344eb0, pyx_v_args=0x7fca523e77c0, 
    pyx_self=<optimized out>) at cupy/util.cpp:2016
#28 pyx_pw_4cupy_4util_7memoize_9decorator_1ret (pyx_self=<optimized out>, pyx_args=0x7fca523e77c0, 
    pyx_kwds=<optimized out>) at cupy/util.cpp:1831
#29 0x00007fca703587f5 in Pyx_PyObject_Call (func=0x7fca7975e050, arg=0x7fca523e77c0, kw=0x7fca52709730)
    at cupy/core/_kernel.cpp:36022
#30 0x00007fca70361fa3 in pyx_f_4cupy_4core_7_kernel_17ElementwiseKernelget_elementwise_kernel (
    pyx_v_self=0x7fca52711d60, pyx_v_dev_id=<optimized out>, pyx_v_arginfos=0x7fca5234b910, 
    pyx_v_type_map=0x7fcaa4af4750, pyx_skip_dispatch=<optimized out>) at cupy/core/_kernel.cpp:17938
#31 0x00007fca70379f9f in pyx_pf_4cupy_4core_7_kernel_17ElementwiseKernel_2call (
    pyx_v_kwargs=0x7fcabe957690, pyx_v_args=0x7fcabe953b50, pyx_v_self=0x7fca52711d60)
    at cupy/core/_kernel.cpp:17297
#32 pyx_pw_4cupy_4core_7_kernel_17ElementwiseKernel_3call (pyx_v_self=0x7fca52711d60, 
    pyx_args=0x7fcabe953b50, __pyx_kwds=<optimized out>) at cupy/core/_kernel.cpp:16373
- 33 0x0000000005700bd4 in _PyObject_FastCallKeywords ()
- 34 0x0000000000c0ca72 in _PyEval_EvalFrameDefault ()
#35 0x00000000057b244c in _PyEval_EvalCodeWithName ()
pyx_v_type_map=0x7fcaa4af4750, pyx_skip_dispatch=<optimized out>) at cupy/core/_kernel.cpp:17938
#31 0x00007fca70379f9f in pyx_pf_4cupy_4core_7_kernel_17ElementwiseKernel_2call (
    pyx_v_kwargs=0x7fcabe957690, pyx_v_args=0x7fcabe953b50, pyx_v_self=0x7fca52711d60)
    at cupy/core/_kernel.cpp:17297
#32 pyx_pw_4cupy_4core_7_kernel_17ElementwiseKernel_3call (pyx_v_self=0x7fca52711d60, 
    pyx_args=0x7fcabe953b50, pyx_kwds=<optimized out>) at cupy/core/_kernel.cpp:16373
- 33 0x0000000005700bd4 in _PyObject_FastCallKeywords ()
- 34 0x0000000000c0ca72 in _PyEval_EvalFrameDefault ()
- 35 0x00000000057b244c in _PyEval_EvalCodeWithName ()
- 36 0x0000000005700536 in _PyFunction_FastCallKeywords ()
- 37 0x0000000000c0dd52 in _PyEval_EvalFrameDefault ()
- 38 0x00000000057b244c in _PyEval_EvalCodeWithName ()
- 39 0x0000000005700536 in _PyFunction_FastCallKeywords ()
- 40 0x0000000000c0e50c in _PyEval_EvalFrameDefault ()
- 41 0x00000000057b244c in _PyEval_EvalCodeWithName ()
- 42 0x00000000057b257e in PyEval_EvalCodeEx ()
- 43 0x00000000057b25ab in PyEval_EvalCode ()
- 44 0x00000000057e57a2 in PyRun_FileExFlags ()
- 45 0x00000000011ec931 in ?? ()
- 46 0x0000000000c10ab8 in _start ()
**System Information** Operating system: Linux-5.3.0-51-generic-x86_64-with-debian-buster-sid 64 Bits Graphics card: GeForce GTX 1050/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 440.59 CUDA: 10.2 **Blender Version** Broken: version: 2.83 (sub 13), branch: master, commit date: 2020-04-13 08:07, hash: `blender/blender@2ec6eca518` Broken: version: 2.83 (sub 15), branch: master, commit date: 2020-05-03 18:42, hash: `blender/blender@331bf04fad` Worked: version: 2.83 (sub 11), branch: master, commit date: 2020-03-31 23:07, hash: `blender/blender@da3cb514e5` **Short description of error** Blender crashes (Segmentation Fault) while trying to use cupy. It will pass the first three flags and then segfault before the fourth. [CudaTest.py](https://archive.blender.org/developer/F8512355/CudaTest.py) **Exact steps for others to reproduce the error** From the command line use `blender --background --python CudaTest.py` This will throw a seg fault If you run the same script outside of blender: $(BLENDER_DIR)/2.83/python/bin/python3.7m CudaTest.py The code will run normally. **Environment setup** Assuming CUDA is installed normally and from a fresh blender install: $(BLENDER_DIR)/2.83/python/bin/python3.7m -m ensurepip $(BLENDER_DIR)/2.83/python/bin/python3.7m -m pip install cupy-cuda102 **Much appreciation** Really appreciate your help! Thanks and hope everyone is staying safe! **Other information** Example backtrace of the core dump below. ```lines=10 - 0 0x000000000d555b20 in nvrtcVersion () - 1 0x00007fca78ca297d in pyx_f_4cupy_4cuda_5nvrtc_getVersion (pyx_skip_dispatch=<optimized out>) at cupy/cuda/nvrtc.cpp:1923 #2 0x00007fca78ca2afb in pyx_pf_4cupy_4cuda_5nvrtc_2getVersion (pyx_self=<optimized out>) at cupy/cuda/nvrtc.cpp:2021 #3 pyx_pw_4cupy_4cuda_5nvrtc_3getVersion (pyx_self=<optimized out>, unused=<optimized out>) at cupy/cuda/nvrtc.cpp:2008 - 4 0x0000000005700a20 in _PyMethodDef_RawFastCallKeywords () - 5 0x0000000005700ab5 in _PyCFunction_FastCallKeywords () - 6 0x0000000000c0d098 in _PyEval_EvalFrameDefault () - 7 0x0000000000c04b80 in ?? () - 8 0x0000000000c0e8fe in _PyEval_EvalFrameDefault () - 9 0x0000000000c04b80 in ?? () - 10 0x0000000000c0e8fe in _PyEval_EvalFrameDefault () - 11 0x00000000057b244c in _PyEval_EvalCodeWithName () - 12 0x0000000005700536 in _PyFunction_FastCallKeywords () - 13 0x0000000000c0e8fe in _PyEval_EvalFrameDefault () - 14 0x00000000057b244c in _PyEval_EvalCodeWithName () - 15 0x0000000005700380 in _PyFunction_FastCallDict () #16 0x00007fcaa487fe57 in __Pyx_PyObject_Call (kw=0x7fca5234b0a0, arg=0x7fca523e7980, func=<optimized out>) at cupy/core/core.cpp:53653 #17 pyx_f_4cupy_4core_4core_compile_with_cache (pyx_v_source=0x7fca5247e1d0, pyx_skip_dispatch=<optimized out>, pyx_optional_args=<optimized out>) at cupy/core/core.cpp:35823 #18 0x00007fca703887a1 in pyx_f_4cupy_4core_7_kernelget_simple_elementwise_kernel ( pyx_v_params=0x7fca527131e0, pyx_v_arginfos=0x7fca5234b910, pyx_v_operation=0x7fca5233d2d0, pyx_v_name=0x7fca5277d490, pyx_v_type_map=<optimized out>, pyx_v_preamble=0x7fcae7c0e3b0, pyx_skip_dispatch=0, pyx_optional_args=0x7ffdf824d010) at cupy/core/_kernel.cpp:4481 #19 0x00007fca70389127 in pyx_pf_4cupy_4core_7_kernel_2_get_simple_elementwise_kernel ( pyx_v_options=<optimized out>, pyx_v_after_loop=<optimized out>, pyx_v_loop_prep=0x7fcae7c0e3b0, pyx_v_preamble=<optimized out>, pyx_v_type_map=<optimized out>, pyx_v_name=0x7fca5277d490, pyx_v_operation=0x7fca5233d2d0, pyx_v_arginfos=0x7fca5234b910, pyx_v_params=0x7fca527131e0, pyx_self=<optimized out>) at cupy/core/_kernel.cpp:4711 #20 pyx_pw_4cupy_4core_7_kernel_3_get_simple_elementwise_kernel (pyx_self=<optimized out>, pyx_args=<optimized out>, pyx_kwds=<optimized out>) at cupy/core/_kernel.cpp:4681 - 21 0x0000000005702705 in PyCFunction_Call () - 22 0x00007fca703860ff in Pyx_PyObject_Call (kw=0x7fca52348500, arg=0x7fca523e78a0, func=0x7fca79748500) at cupy/core/_kernel.cpp:36022 #23 pyx_pf_4cupy_4core_7_kernel_12_get_elementwise_kernel (pyx_v_kwargs=0x7fca52348500, pyx_v_preamble=0x7fcae7c0e3b0, pyx_v_name=0x7fca5277d490, pyx_v_operation=0x7fca5233d2d0, pyx_v_params=0x7fca527131e0, pyx_v_type_map=0x7fcaa4af4750, pyx_v_arginfos=0x7fca5234b910, pyx_self=<optimized out>) at cupy/core/_kernel.cpp:15617 #24 pyx_pw_4cupy_4core_7_kernel_13_get_elementwise_kernel (__pyx_self=<optimized out>, pyx_args=<optimized out>, pyx_kwds=<optimized out>) at cupy/core/_kernel.cpp:15281 - 25 0x0000000005702705 in PyCFunction_Call () - 26 0x00007fca862468a7 in Pyx_PyObject_Call (kw=0x7fca52344eb0, arg=0x7fca523e77c0, func=0x7fca7975a410) at cupy/util.cpp:4384 #27 pyx_pf_4cupy_4util_7memoize_9decorator_ret (pyx_v_kwargs=0x7fca52344eb0, pyx_v_args=0x7fca523e77c0, pyx_self=<optimized out>) at cupy/util.cpp:2016 #28 pyx_pw_4cupy_4util_7memoize_9decorator_1ret (pyx_self=<optimized out>, pyx_args=0x7fca523e77c0, pyx_kwds=<optimized out>) at cupy/util.cpp:1831 #29 0x00007fca703587f5 in Pyx_PyObject_Call (func=0x7fca7975e050, arg=0x7fca523e77c0, kw=0x7fca52709730) at cupy/core/_kernel.cpp:36022 #30 0x00007fca70361fa3 in pyx_f_4cupy_4core_7_kernel_17ElementwiseKernelget_elementwise_kernel ( pyx_v_self=0x7fca52711d60, pyx_v_dev_id=<optimized out>, pyx_v_arginfos=0x7fca5234b910, pyx_v_type_map=0x7fcaa4af4750, pyx_skip_dispatch=<optimized out>) at cupy/core/_kernel.cpp:17938 #31 0x00007fca70379f9f in pyx_pf_4cupy_4core_7_kernel_17ElementwiseKernel_2call ( pyx_v_kwargs=0x7fcabe957690, pyx_v_args=0x7fcabe953b50, pyx_v_self=0x7fca52711d60) at cupy/core/_kernel.cpp:17297 #32 pyx_pw_4cupy_4core_7_kernel_17ElementwiseKernel_3call (pyx_v_self=0x7fca52711d60, pyx_args=0x7fcabe953b50, __pyx_kwds=<optimized out>) at cupy/core/_kernel.cpp:16373 - 33 0x0000000005700bd4 in _PyObject_FastCallKeywords () - 34 0x0000000000c0ca72 in _PyEval_EvalFrameDefault () #35 0x00000000057b244c in _PyEval_EvalCodeWithName () pyx_v_type_map=0x7fcaa4af4750, pyx_skip_dispatch=<optimized out>) at cupy/core/_kernel.cpp:17938 #31 0x00007fca70379f9f in pyx_pf_4cupy_4core_7_kernel_17ElementwiseKernel_2call ( pyx_v_kwargs=0x7fcabe957690, pyx_v_args=0x7fcabe953b50, pyx_v_self=0x7fca52711d60) at cupy/core/_kernel.cpp:17297 #32 pyx_pw_4cupy_4core_7_kernel_17ElementwiseKernel_3call (pyx_v_self=0x7fca52711d60, pyx_args=0x7fcabe953b50, pyx_kwds=<optimized out>) at cupy/core/_kernel.cpp:16373 - 33 0x0000000005700bd4 in _PyObject_FastCallKeywords () - 34 0x0000000000c0ca72 in _PyEval_EvalFrameDefault () - 35 0x00000000057b244c in _PyEval_EvalCodeWithName () - 36 0x0000000005700536 in _PyFunction_FastCallKeywords () - 37 0x0000000000c0dd52 in _PyEval_EvalFrameDefault () - 38 0x00000000057b244c in _PyEval_EvalCodeWithName () - 39 0x0000000005700536 in _PyFunction_FastCallKeywords () - 40 0x0000000000c0e50c in _PyEval_EvalFrameDefault () - 41 0x00000000057b244c in _PyEval_EvalCodeWithName () - 42 0x00000000057b257e in PyEval_EvalCodeEx () - 43 0x00000000057b25ab in PyEval_EvalCode () - 44 0x00000000057e57a2 in PyRun_FileExFlags () - 45 0x00000000011ec931 in ?? () - 46 0x0000000000c10ab8 in _start () ```
Author

Added subscriber: @nkundtz

Added subscriber: @nkundtz

Added subscriber: @lvxejay

Added subscriber: @lvxejay
Member

Added subscriber: @ankitm

Added subscriber: @ankitm

Looks like the bug might have something to do with the symbol visibility changes introduced in bae1c243cec: https://developer.blender.org/rBbae1c243cec809ac783c1b8f2fb65a570b526ef0

I tested the previous commit, f1573731bc8, which works as expected, no segfault: https://developer.blender.org/rBf1573731bc8b80c03ef5db4292568781969846c3

Looks like the bug might have something to do with the symbol visibility changes introduced in `bae1c243cec`: https://developer.blender.org/rBbae1c243cec809ac783c1b8f2fb65a570b526ef0 I tested the previous commit, `f1573731bc8`, which works as expected, no segfault: https://developer.blender.org/rBf1573731bc8b80c03ef5db4292568781969846c3

Added subscriber: @brecht

Added subscriber: @brecht

Changed status from 'Needs Triage' to: 'Needs User Info'

Changed status from 'Needs Triage' to: 'Needs User Info'

Does this work in 2.82?

What I'm guessing is that this briefly worked after blender/blender@b555b8dedc, then part of that got reverted in blender/blender@bae1c243ce and it stopped working again?

Does this work in 2.82? What I'm guessing is that this briefly worked after blender/blender@b555b8dedc, then part of that got reverted in blender/blender@bae1c243ce and it stopped working again?
Author

Based on my testing and that of the NVIDIA team Segmentation faults are produced in all of our tested 2.82, 2.79, or alpha 2.90 releases.

Based on my testing and that of the NVIDIA team Segmentation faults are produced in all of our tested 2.82, 2.79, or alpha 2.90 releases.

This issue was referenced by blender/blender@9d8a583482

This issue was referenced by blender/blender@9d8a5834829c3a30048dffde2346878f71cbd5fb

Changed status from 'Needs User Info' to: 'Resolved'

Changed status from 'Needs User Info' to: 'Resolved'
Brecht Van Lommel self-assigned this 2020-05-05 16:03:34 +02:00
Sign in to join this conversation.
No Milestone
No project
No Assignees
5 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender-addons#76414
No description provided.