Crash when running python script from command line #85141

New Issue

Netherby · 2021-01-28T07:13:55+01:00

Netherby commented

2021-01-28 07:13:55 +01:00

System Information
Operating system: Windows 10
Graphics card: Nvidia 1080

Blender Version
Broken: 2.91.2, Commit date: 2021-01-19 16:15, Hash 5be9ef4177
Worked: 2.91.0

Short description of error
Crashes in random locations when running a python script from command line. Script sometimes completes, other times crashes with no error message and finally sometimes will generate a crash log with EXCEPTION_ACCESS_VIOLATION.

Thread that crashes is not the one executing the script. However changing bpy.context.window.scene, calling bpy.context.view_layer.update() or using other functions that cause an update are required to trigger the crash (as far as I can tell). Very hard to debug as it seems like a blender internal thread is trying to free a null reference at some point when it feels like it.

I'm including a crash log from when I can actually get it to make one.
BW_BWTest.crash.txt

See comments for steps to reproduce.

**System Information** Operating system: Windows 10 Graphics card: Nvidia 1080 **Blender Version** Broken: 2.91.2, Commit date: 2021-01-19 16:15, Hash 5be9ef417703 Worked: 2.91.0 **Short description of error** Crashes in random locations when running a python script from command line. Script sometimes completes, other times crashes with no error message and finally sometimes will generate a crash log with EXCEPTION_ACCESS_VIOLATION. Thread that crashes is not the one executing the script. However changing bpy.context.window.scene, calling bpy.context.view_layer.update() or using other functions that cause an update are required to trigger the crash (as far as I can tell). Very hard to debug as it seems like a blender internal thread is trying to free a null reference at some point when it feels like it. I'm including a crash log from when I can actually get it to make one. [BW_BWTest.crash.txt](https://archive.blender.org/developer/F9602138/BW_BWTest.crash.txt) See comments for steps to reproduce.

Netherby commented

2021-01-28 07:13:55 +01:00

Added subscriber: @JerahmyPocott

Campbell Barton commented

2021-01-28 08:15:29 +01:00

Added subscriber: @ideasman42

Campbell Barton commented

2021-01-28 08:15:29 +01:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Campbell Barton commented

2021-01-28 08:15:29 +01:00

This report is missing:

Exact steps to redo the error.
A blend file (unless there are simple steps to redo this from the default startup).

This report is missing: - Exact steps to redo the error. - A blend file (unless there are simple steps to redo this from the default startup).

Robert Guetzkow commented

2021-01-28 09:47:23 +01:00

Added subscriber: @rjg

Robert Guetzkow commented

2021-01-28 09:47:23 +01:00

We would also need the script that causes the crash otherwise we won't be able to reproduce the same conditions you had that resulted in the crash.

Netherby commented

2021-01-28 10:35:37 +01:00

The problem is that there is not a guaranteed way to reproduce it. Seems likely to be a race condition resulting in a null reference being freed, which I hoped you could determine from the crash log.

I have a script that will cause it with a high degree of frequency on my system, but it is part of a paid add-on. You would need to whole add-on to use it.

I can try to create a script to trigger it... (Building a debug version of the release is also an option, but that will take a while)

The problem is that there is not a guaranteed way to reproduce it. Seems likely to be a race condition resulting in a null reference being freed, which I hoped you could determine from the crash log. I have a script that will cause it with a high degree of frequency on my system, but it is part of a paid add-on. You would need to whole add-on to use it. I can try to create a script to trigger it... (Building a debug version of the release is also an option, but that will take a while)

Netherby commented

2021-01-28 11:40:45 +01:00

Okay I think I've narrowed the problem down to using bpy.data.libraries.load to import things from a library blend file. I have created a library.blend and a separate blend with a few scenes and objects plus a python script which will load some stuff from the library and then try to bake emit passes of all the objects in the other file after performing some manipulations.

With none of the library loading code, nothing goes wrong in my tests. With the library objects being loaded it will crash a large proportion of the time (but not every single time) either silently or sometimes with error message and log.

Please unzip the files into one place and run the blender 2.91.2 version with the options: BWTest.blend --background --python makecrash.py

You may need to run it more than once in order to produce the crash.
crash.zip

Okay I think I've narrowed the problem down to using bpy.data.libraries.load to import things from a library blend file. I have created a library.blend and a separate blend with a few scenes and objects plus a python script which will load some stuff from the library and then try to bake emit passes of all the objects in the other file after performing some manipulations. With none of the library loading code, nothing goes wrong in my tests. With the library objects being loaded it will crash a large proportion of the time (but not every single time) either silently or sometimes with error message and log. Please unzip the files into one place and run the blender 2.91.2 version with the options: BWTest.blend --background --python makecrash.py You may need to run it more than once in order to produce the crash. [crash.zip](https://archive.blender.org/developer/F9602371/crash.zip)

Robert Guetzkow commented

2021-01-28 13:56:17 +01:00

Changed status from 'Needs User Info' to: 'Needs Triage'

Robert Guetzkow commented

2021-01-28 14:52:37 +01:00

There is a shift by a negative number happening in get_levels_from_disps (multires.c) for int side = (1 << (totlvl - 1)) + 1;, followed by an integer overflow when using the result in:

int lvl_totdisp = side * side;

This results in any entirely different stack trace than the one you've attached in your crash log, but given that this causes undefined behavior it may crash for you at a later point in time.

T85141_even_simpler.blend

The integer overflow happens when creating a copy of the object and its data.

There is a shift by a negative number happening in `get_levels_from_disps` (`multires.c`) for `int side = (1 << (totlvl - 1)) + 1;`, followed by an integer overflow when using the result in: ``` int lvl_totdisp = side * side; ``` This results in any entirely different stack trace than the one you've attached in your crash log, but given that this causes undefined behavior it may crash for you at a later point in time. [T85141_even_simpler.blend](https://archive.blender.org/developer/F9602580/T85141_even_simpler.blend) The integer overflow happens when creating a copy of the object and its data.

Robert Guetzkow commented

2021-01-28 14:54:06 +01:00

Given the complexity of your project I hope that is the same issue that you've been experiencing. Please try to see if you can reduce the problem down to the passtest object on your system as well.

Given the complexity of your project I hope that is the same issue that you've been experiencing. Please try to see if you can reduce the problem down to the `passtest` object on your system as well.

Robert Guetzkow commented

2021-01-28 14:54:47 +01:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Netherby commented

2021-01-29 05:13:24 +01:00

I don't believe that is my issue (or at least not the only issue).

I have further reduced the script and the necessary steps I'm seeing are:

use bpy.data.libraries.load to add a scene
create a new scene and set the context to the newly created scene
link some objects into the new scene
cause the scene to update (I'm calling view_layer.update a bunch of times)

Here is my greatly reduced script (still using the library and blend file from previous)
makecrash2.py

I don't believe that is my issue (or at least not the only issue). I have further reduced the script and the necessary steps I'm seeing are: * use bpy.data.libraries.load to add a scene * create a new scene and set the context to the newly created scene * link some objects into the new scene * cause the scene to update (I'm calling view_layer.update a bunch of times) Here is my greatly reduced script (still using the library and blend file from previous) [makecrash2.py](https://archive.blender.org/developer/F9603450/makecrash2.py)

Campbell Barton commented

2021-01-29 11:19:22 +01:00

Cant redo:

OS: Linux 5.10.10-arch1-1
REV: e4776fa4e7993b93f358748c56427ccb5d1b75f4
GPU: [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT]

Running this command:

blender.bin -b T85141_even_simpler.blend --python makecrash2.py

Outputs:

Read blend: /d/crash/T85141_even_simpler.blend

--------------------------------------------------------------
---Complete                                                ---


Blender quit

Cant redo: * OS: Linux 5.10.10-arch1-1 * REV: `e4776fa4e7993b93f358748c56427ccb5d1b75f4` * GPU: `[AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT]` ---- Running this command: `blender.bin -b T85141_even_simpler.blend --python makecrash2.py` Outputs: ``` Read blend: /d/crash/T85141_even_simpler.blend -------------------------------------------------------------- ---Complete --- Blender quit ```

Netherby commented

2021-01-29 13:59:31 +01:00

How many times did you run it? The crash does not happen 100% of the time... But if you run it enough times it should trigger, it does for me. (Unless of course it doesn't affect linux build)

Sometimes I can run it like 10+ times in a row with no error, but eventually it does crash.

How many times did you run it? The crash does not happen 100% of the time... But if you run it enough times it should trigger, it does for me. (Unless of course it doesn't affect linux build) Sometimes I can run it like 10+ times in a row with no error, but eventually it does crash.

Robert Guetzkow commented

2021-01-29 14:25:55 +01:00

@JerahmyPocott Running it with the original BWTest.blend using ./blender -b ~/01-data/03-bug-tracker/crash/BWTest.blend --python ~/01-data/03-bug-tracker/crash/makecrash2.py

It crashes in 2.91.2 (release build) and seems to work fine a debug build of 2.91.0
Appears to work fine in 2.92.0 and 2.93.0 (release build and debug build with ASAN)

Stack trace in 2.91.2:

# backtrace
./blender(BLI_system_backtrace+0x20) [0x84bbb90]
./blender() [0xe529bd]
/lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fdbab099210]
./blender() [0x111bd30]
./blender(makeDerivedMesh+0x4c) [0x111ed9c]
./blender(BKE_object_handle_data_update+0x3a0) [0xee43f0]
./blender(BKE_object_eval_uber_data+0x2f) [0xee46ff]
./blender() [0x710d432]
./blender() [0x710d602]
./blender() [0x11acfe5]
./blender() [0x84bf7d3]
./blender() [0x11ba6e5]
./blender() [0x11ba99b]
./blender() [0x11accf7]
./blender() [0x11b4460]
./blender() [0x11b646c]
./blender() [0x11b6669]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fdbab795609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fdbab175293]

I'm compiling a debug build of 2.91.2 with ASAN now. It could be that the bug has already been fixed or it isn't as easy to trigger in more recent versions.

@JerahmyPocott Running it with the original `BWTest.blend` using `./blender -b ~/01-data/03-bug-tracker/crash/BWTest.blend --python ~/01-data/03-bug-tracker/crash/makecrash2.py` - It crashes in 2.91.2 (release build) and seems to work fine a debug build of 2.91.0 - Appears to work fine in 2.92.0 and 2.93.0 (release build and debug build with ASAN) Stack trace in 2.91.2: ```lines # backtrace ./blender(BLI_system_backtrace+0x20) [0x84bbb90] ./blender() [0xe529bd] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fdbab099210] ./blender() [0x111bd30] ./blender(makeDerivedMesh+0x4c) [0x111ed9c] ./blender(BKE_object_handle_data_update+0x3a0) [0xee43f0] ./blender(BKE_object_eval_uber_data+0x2f) [0xee46ff] ./blender() [0x710d432] ./blender() [0x710d602] ./blender() [0x11acfe5] ./blender() [0x84bf7d3] ./blender() [0x11ba6e5] ./blender() [0x11ba99b] ./blender() [0x11accf7] ./blender() [0x11b4460] ./blender() [0x11b646c] ./blender() [0x11b6669] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fdbab795609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fdbab175293] ``` I'm compiling a debug build of 2.91.2 with ASAN now. It could be that the bug has already been fixed or it isn't as easy to trigger in more recent versions.

Robert Guetzkow commented

2021-01-29 15:40:36 +01:00

The assert BLI_assert(id->session_uuid != MAIN_ID_SESSION_UUID_UNSET); in DepsgraphNodeBuilder::add_id_node(ID *id) is triggered in 2.91.2.

libc.so.6!__GI_raise(int sig) (/build/glibc-ZN95T4/glibc-2.31/sysdeps/unix/sysv/linux/raise.c:50)
libc.so.6!__GI_abort() (/build/glibc-ZN95T4/glibc-2.31/stdlib/abort.c:79)
_BLI_assert_abort() (/home/dev/01-data/01-git/blender-git/blender/source/blender/blenlib/intern/BLI_assert.c:50)
blender::deg::DepsgraphNodeBuilder::add_id_node(blender::deg::DepsgraphNodeBuilder * const this, ID * id) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/deg_builder_nodes.cc:156)
blender::deg::DepsgraphNodeBuilder::build_collection(blender::deg::DepsgraphNodeBuilder * const this, LayerCollection * from_layer_collection, Collection * collection) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/deg_builder_nodes.cc:526)
blender::deg::DepsgraphNodeBuilder::build_layer_collections(blender::deg::DepsgraphNodeBuilder * const this, ListBase * lb) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/deg_builder_nodes_view_layer.cc:71)
blender::deg::DepsgraphNodeBuilder::build_view_layer(blender::deg::DepsgraphNodeBuilder * const this, Scene * scene, ViewLayer * view_layer, blender::deg::eDepsNode_LinkedState_Type linked_state) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/deg_builder_nodes_view_layer.cc:121)
blender::deg::ViewLayerBuilderPipeline::build_nodes(blender::deg::ViewLayerBuilderPipeline * const this, blender::deg::DepsgraphNodeBuilder & node_builder) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/pipeline_view_layer.cc:36)
blender::deg::AbstractBuilderPipeline::build_step_nodes(blender::deg::AbstractBuilderPipeline * const this) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/pipeline.cc:78)
blender::deg::AbstractBuilderPipeline::build(blender::deg::AbstractBuilderPipeline * const this) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/pipeline.cc:57)
DEG_graph_build_from_view_layer(Depsgraph * graph) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/depsgraph_build.cc:216)
DEG_graph_relations_update(Depsgraph * graph) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/depsgraph_build.cc:269)
ED_scene_change_update(Main * bmain, Scene * scene, ViewLayer * layer) (/home/dev/01-data/01-git/blender-git/blender/source/blender/editors/scene/scene_edit.c:122)
WM_window_set_active_scene(Main * bmain, bContext * C, wmWindow * win, Scene * scene) (/home/dev/01-data/01-git/blender-git/blender/source/blender/windowmanager/intern/wm_window.c:2304)
rna_Window_scene_update(bContext * C, PointerRNA * ptr) (/home/dev/01-data/01-git/blender-git/blender/source/blender/makesrna/intern/rna_wm.c:732)
rna_property_update(bContext * C, Main * bmain, Scene * scene, PointerRNA * ptr, PropertyRNA * prop) (/home/dev/01-data/01-git/blender-git/blender/source/blender/makesrna/intern/rna_access.c:2244)
RNA_property_update(bContext * C, PointerRNA * ptr, PropertyRNA * prop) (/home/dev/01-data/01-git/blender-git/blender/source/blender/makesrna/intern/rna_access.c:2315)
pyrna_py_to_prop(PointerRNA * ptr, PropertyRNA * prop, void * data, PyObject * value, const char * error_prefix) (/home/dev/01-data/01-git/blender-git/blender/source/blender/python/intern/bpy_rna.c:2220)
pyrna_struct_setattro(BPy_StructRNA * self, PyObject * pyname, PyObject * value) (/home/dev/01-data/01-git/blender-git/blender/source/blender/python/intern/bpy_rna.c:4518)
PyObject_SetAttr(PyObject * v, PyObject * name, PyObject * value) (/home/blender/Developer/build_linux/deps/build/python/src/external_python/Objects/object.c:1017)

The assert `BLI_assert(id->session_uuid != MAIN_ID_SESSION_UUID_UNSET);` in `DepsgraphNodeBuilder::add_id_node(ID *id)` is triggered in 2.91.2. ```lines libc.so.6!__GI_raise(int sig) (/build/glibc-ZN95T4/glibc-2.31/sysdeps/unix/sysv/linux/raise.c:50) libc.so.6!__GI_abort() (/build/glibc-ZN95T4/glibc-2.31/stdlib/abort.c:79) _BLI_assert_abort() (/home/dev/01-data/01-git/blender-git/blender/source/blender/blenlib/intern/BLI_assert.c:50) blender::deg::DepsgraphNodeBuilder::add_id_node(blender::deg::DepsgraphNodeBuilder * const this, ID * id) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/deg_builder_nodes.cc:156) blender::deg::DepsgraphNodeBuilder::build_collection(blender::deg::DepsgraphNodeBuilder * const this, LayerCollection * from_layer_collection, Collection * collection) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/deg_builder_nodes.cc:526) blender::deg::DepsgraphNodeBuilder::build_layer_collections(blender::deg::DepsgraphNodeBuilder * const this, ListBase * lb) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/deg_builder_nodes_view_layer.cc:71) blender::deg::DepsgraphNodeBuilder::build_view_layer(blender::deg::DepsgraphNodeBuilder * const this, Scene * scene, ViewLayer * view_layer, blender::deg::eDepsNode_LinkedState_Type linked_state) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/deg_builder_nodes_view_layer.cc:121) blender::deg::ViewLayerBuilderPipeline::build_nodes(blender::deg::ViewLayerBuilderPipeline * const this, blender::deg::DepsgraphNodeBuilder & node_builder) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/pipeline_view_layer.cc:36) blender::deg::AbstractBuilderPipeline::build_step_nodes(blender::deg::AbstractBuilderPipeline * const this) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/pipeline.cc:78) blender::deg::AbstractBuilderPipeline::build(blender::deg::AbstractBuilderPipeline * const this) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/builder/pipeline.cc:57) DEG_graph_build_from_view_layer(Depsgraph * graph) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/depsgraph_build.cc:216) DEG_graph_relations_update(Depsgraph * graph) (/home/dev/01-data/01-git/blender-git/blender/source/blender/depsgraph/intern/depsgraph_build.cc:269) ED_scene_change_update(Main * bmain, Scene * scene, ViewLayer * layer) (/home/dev/01-data/01-git/blender-git/blender/source/blender/editors/scene/scene_edit.c:122) WM_window_set_active_scene(Main * bmain, bContext * C, wmWindow * win, Scene * scene) (/home/dev/01-data/01-git/blender-git/blender/source/blender/windowmanager/intern/wm_window.c:2304) rna_Window_scene_update(bContext * C, PointerRNA * ptr) (/home/dev/01-data/01-git/blender-git/blender/source/blender/makesrna/intern/rna_wm.c:732) rna_property_update(bContext * C, Main * bmain, Scene * scene, PointerRNA * ptr, PropertyRNA * prop) (/home/dev/01-data/01-git/blender-git/blender/source/blender/makesrna/intern/rna_access.c:2244) RNA_property_update(bContext * C, PointerRNA * ptr, PropertyRNA * prop) (/home/dev/01-data/01-git/blender-git/blender/source/blender/makesrna/intern/rna_access.c:2315) pyrna_py_to_prop(PointerRNA * ptr, PropertyRNA * prop, void * data, PyObject * value, const char * error_prefix) (/home/dev/01-data/01-git/blender-git/blender/source/blender/python/intern/bpy_rna.c:2220) pyrna_struct_setattro(BPy_StructRNA * self, PyObject * pyname, PyObject * value) (/home/dev/01-data/01-git/blender-git/blender/source/blender/python/intern/bpy_rna.c:4518) PyObject_SetAttr(PyObject * v, PyObject * name, PyObject * value) (/home/blender/Developer/build_linux/deps/build/python/src/external_python/Objects/object.c:1017) ```

Netherby commented

2021-01-30 03:29:42 +01:00

In #85141#1103478, @rjg wrote:
@JerahmyPocott Running it with the original BWTest.blend using ./blender -b ~/01-data/03-bug-tracker/crash/BWTest.blend --python ~/01-data/03-bug-tracker/crash/makecrash2.py

It crashes in 2.91.2 (release build) and seems to work fine a debug build of 2.91.0

Appears to work fine in 2.92.0 and 2.93.0 (release build and debug build with ASAN)

As far as I can tell it works fine in 2.91.0 for me (or triggers so rarely I haven't seen it).

> In #85141#1103478, @rjg wrote: > @JerahmyPocott Running it with the original `BWTest.blend` using `./blender -b ~/01-data/03-bug-tracker/crash/BWTest.blend --python ~/01-data/03-bug-tracker/crash/makecrash2.py` > > - It crashes in 2.91.2 (release build) and seems to work fine a debug build of 2.91.0 > - Appears to work fine in 2.92.0 and 2.93.0 (release build and debug build with ASAN) As far as I can tell it works fine in 2.91.0 for me (or triggers so rarely I haven't seen it).

Netherby commented

2021-02-03 13:50:22 +01:00

Appears to be working fine in 2.92 for me using 3 Feb build.

Campbell Barton commented

2021-02-04 03:39:49 +01:00

Changed status from 'Needs User Info' to: 'Resolved'

Campbell Barton closed this issue

2021-02-04 03:39:49 +01:00

Campbell Barton self-assigned this 2021-02-04 03:39:49 +01:00

Campbell Barton commented

2021-02-04 03:39:49 +01:00

Closing as this seems to be resolved.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Crash when running python script from command line #85141