Render crash when using Python API to modify object data in frame_change_pre handler #60094

Closed
opened 2019-01-03 00:27:43 +01:00 by Ryan Guy · 77 comments

System Information
Operating system: Windows 10 Home
Graphics card: GTX 1070 8GB

Blender Version
Broken: 2.80, .4dd0a90f4213, 2019-01-01
Worked: 2.79b release

Short description of error

Blender crashes while rendering and animation when object data is modified inside the frame_change_pre handler using the Python API. The crashes only seem to occur while rendering and not during playback. The frequency of crashes seems to be related to the frequency at which the object data is modified.

This is part of an addon that reads and renders geometry data from a sequence of files. The addon does this by creating a Blender object, and on each frame swap out the object's mesh data with new geometry and then delete the old geometry. I have a created a simplified script that reproduces this crash within the first few frames of rendering:

import bpy


def frame_change_pre(scene):
    # The addon would load geometry from a file, but for a simplified test an icosphere works
    vertices = [
        (0.0000, 0.0000, -1.0000), (0.7236, -0.5257, -0.4472), (-0.2764, -0.8506, -0.4472), (-0.8944, 0.0000, -0.4472),
        (-0.2764, 0.8506, -0.4472), (0.7236, 0.5257, -0.4472), (0.2764, -0.8506, 0.4472), (-0.7236, -0.5257, 0.4472),
        (-0.7236, 0.5257, 0.4472), (0.2764, 0.8506, 0.4472), (0.8944, 0.0000, 0.4472), (0.0000, 0.0000, 1.0000)
    ]

    triangles = [
        (0, 1, 2), (1, 0, 5), (0, 2, 3), (0, 3, 4), (0, 4, 5), (1, 5, 10), (2, 1, 6), (3, 2, 7), (4, 3, 8), (5, 4, 9),
        (1, 10, 6), (2, 6, 7), (3, 7, 8), (4, 8, 9), (5, 9, 10), (6, 10, 11), (7, 6, 11), (8, 7, 11), (9, 8, 11), (10, 9, 11)
    ]
    
    # Create a new mesh with geometry
    new_mesh_data = bpy.data.meshes.new("mesh_data" + str(scene.frame_current))
    new_mesh_data.from_pydata(vertices, [], triangles)
    
    # Swap the new mesh data and delete the old mesh data
    mesh_cache = bpy.data.objects.get("mesh_cache")
    old_mesh_data = mesh_cache.data
    mesh_cache.data = new_mesh_data
    bpy.data.meshes.remove(old_mesh_data)
    
    - This is what causes the crash: the more frequently the mesh cache data is accessed, the more frequent the crash occurs.
    - For a simplified test, we will repeatedly set smooth shading on the mesh data polygons. This also happens if setting the 
    # object location/scale/matrix_world, mesh data materials, and other data.
    for i in range(1000):
        for p in mesh_cache.data.polygons:
            p.use_smooth = True
    

# Create a cache object to store the current frame mesh    
mesh_cache_data = bpy.data.meshes.new("mesh_cache_data")
mesh_cache_data.from_pydata([], [], [])
mesh_cache_object = bpy.data.objects.new("mesh_cache", mesh_cache_data)
bpy.context.scene.collection.objects.link(mesh_cache_object)

bpy.app.handlers.frame_change_pre.append(frame_change_pre)

This is the error Blender reports (EXCEPTION_ACCESS_VIOLATION):

Error   : EXCEPTION_ACCESS_VIOLATION
Address : 0x00007FF6F95E6650
Module  : C:\\Users\\ryanl\\Downloads\\blender-2.80.0-git.4dd0a90f4213-windows64\\blender.exe

Exact steps for others to reproduce the error

Attached is a .blend file including the script that reproduces this issue.

  1. Open the .blend file
  2. Press 'Run Script'
  3. Begin rendering the animation (Blender > Render > Render Animation)

bug_script_280.blend

**System Information** Operating system: Windows 10 Home Graphics card: GTX 1070 8GB **Blender Version** Broken: 2.80, .4dd0a90f4213, 2019-01-01 Worked: 2.79b release **Short description of error** Blender crashes while rendering and animation when object data is modified inside the frame_change_pre handler using the Python API. The crashes only seem to occur while rendering and not during playback. The frequency of crashes seems to be related to the frequency at which the object data is modified. This is part of an addon that reads and renders geometry data from a sequence of files. The addon does this by creating a Blender object, and on each frame swap out the object's mesh data with new geometry and then delete the old geometry. I have a created a simplified script that reproduces this crash within the first few frames of rendering: ``` import bpy def frame_change_pre(scene): # The addon would load geometry from a file, but for a simplified test an icosphere works vertices = [ (0.0000, 0.0000, -1.0000), (0.7236, -0.5257, -0.4472), (-0.2764, -0.8506, -0.4472), (-0.8944, 0.0000, -0.4472), (-0.2764, 0.8506, -0.4472), (0.7236, 0.5257, -0.4472), (0.2764, -0.8506, 0.4472), (-0.7236, -0.5257, 0.4472), (-0.7236, 0.5257, 0.4472), (0.2764, 0.8506, 0.4472), (0.8944, 0.0000, 0.4472), (0.0000, 0.0000, 1.0000) ] triangles = [ (0, 1, 2), (1, 0, 5), (0, 2, 3), (0, 3, 4), (0, 4, 5), (1, 5, 10), (2, 1, 6), (3, 2, 7), (4, 3, 8), (5, 4, 9), (1, 10, 6), (2, 6, 7), (3, 7, 8), (4, 8, 9), (5, 9, 10), (6, 10, 11), (7, 6, 11), (8, 7, 11), (9, 8, 11), (10, 9, 11) ] # Create a new mesh with geometry new_mesh_data = bpy.data.meshes.new("mesh_data" + str(scene.frame_current)) new_mesh_data.from_pydata(vertices, [], triangles) # Swap the new mesh data and delete the old mesh data mesh_cache = bpy.data.objects.get("mesh_cache") old_mesh_data = mesh_cache.data mesh_cache.data = new_mesh_data bpy.data.meshes.remove(old_mesh_data) - This is what causes the crash: the more frequently the mesh cache data is accessed, the more frequent the crash occurs. - For a simplified test, we will repeatedly set smooth shading on the mesh data polygons. This also happens if setting the # object location/scale/matrix_world, mesh data materials, and other data. for i in range(1000): for p in mesh_cache.data.polygons: p.use_smooth = True # Create a cache object to store the current frame mesh mesh_cache_data = bpy.data.meshes.new("mesh_cache_data") mesh_cache_data.from_pydata([], [], []) mesh_cache_object = bpy.data.objects.new("mesh_cache", mesh_cache_data) bpy.context.scene.collection.objects.link(mesh_cache_object) bpy.app.handlers.frame_change_pre.append(frame_change_pre) ``` This is the error Blender reports (**EXCEPTION_ACCESS_VIOLATION**): ``` Error : EXCEPTION_ACCESS_VIOLATION Address : 0x00007FF6F95E6650 Module : C:\\Users\\ryanl\\Downloads\\blender-2.80.0-git.4dd0a90f4213-windows64\\blender.exe ``` **Exact steps for others to reproduce the error** Attached is a .blend file including the script that reproduces this issue. 1. Open the .blend file 2. Press 'Run Script' 3. Begin rendering the animation (Blender > Render > Render Animation) [bug_script_280.blend](https://archive.blender.org/developer/F6154836/bug_script_280.blend)
Author

Added subscriber: @rlguy

Added subscriber: @rlguy

blender/blender#67627 was marked as duplicate of this issue

blender/blender#67627 was marked as duplicate of this issue

blender/blender#67819 was marked as duplicate of this issue

blender/blender#67819 was marked as duplicate of this issue

blender/blender#64065 was marked as duplicate of this issue

blender/blender#64065 was marked as duplicate of this issue

blender/blender#63610 was marked as duplicate of this issue

blender/blender#63610 was marked as duplicate of this issue

blender/blender#62632 was marked as duplicate of this issue

blender/blender#62632 was marked as duplicate of this issue

#62096 was marked as duplicate of this issue

#62096 was marked as duplicate of this issue

Added subscriber: @Klutz

Added subscriber: @Klutz

Can confirm the same bug on the Linux build:

version: 2.80 (sub 40), branch: blender2.7, commit date: 2019-01-04 21:18, hash: a77b63c56943, type: Release
build date: 2019-01-05, 00:56:39
platform: Linux

Can confirm the same bug on the Linux build: version: 2.80 (sub 40), branch: blender2.7, commit date: 2019-01-04 21:18, hash: a77b63c56943, type: Release build date: 2019-01-05, 00:56:39 platform: Linux
Campbell Barton was assigned by Sebastian Parborg 2019-01-25 11:36:36 +01:00

Added subscriber: @AdrianCabrera

Added subscriber: @AdrianCabrera

Added subscriber: @knusk

Added subscriber: @knusk

Added subscribers: @Sergey, @ideasman42, @brecht

Added subscribers: @Sergey, @ideasman42, @brecht
Campbell Barton was unassigned by Brecht Van Lommel 2019-03-18 14:00:25 +01:00

Changing priority since variations of this crash have been reported a few times, and commonly used add-ons like Animation Nodes and FLIP fluids suffer from it.

This may be more a dependency graph issue than a Python API one.

In general it would be good to clarify which kind of data scripts are allowed or recommended to edit in the various handlers, it's not so clear anymore with multiple depsgraphs.

Changing priority since variations of this crash have been reported a few times, and commonly used add-ons like Animation Nodes and FLIP fluids suffer from it. This may be more a dependency graph issue than a Python API one. In general it would be good to clarify which kind of data scripts are allowed or recommended to edit in the various handlers, it's not so clear anymore with multiple depsgraphs.
Member

Added subscriber: @JacquesLucke

Added subscriber: @JacquesLucke
Member

Added subscriber: @OmarEmaraDev

Added subscriber: @OmarEmaraDev

Added subscriber: @anon

Added subscriber: @anon

Added subscriber: @dsine

Added subscriber: @dsine

Added subscriber: @juang3d

Added subscriber: @juang3d

Just stumbled upon this bug, what a big bug!

I'm being completely unable to render one project right now because of this, I hope it gets fixed soon.

Cheers!

Just stumbled upon this bug, what a big bug! I'm being completely unable to render one project right now because of this, I hope it gets fixed soon. Cheers!

Added subscriber: @iad

Added subscriber: @iad

Added subscriber: @hansfbaier

Added subscriber: @hansfbaier

Added subscriber: @bestdani

Added subscriber: @bestdani
Member

Added subscribers: @EricS-4, @LazyDodo, @1D_Inc

Added subscribers: @EricS-4, @LazyDodo, @1D_Inc

Added subscriber: @MichaelHermann

Added subscriber: @MichaelHermann
Sergey Sharybin was assigned by Brecht Van Lommel 2019-05-28 17:43:25 +02:00

Developer note: this issue is caused by a threading conflict between viewport which is catching with its dependency graph in the main thread with the render thread which is modifying main database while it is used by viewport.

Not sure how/whether this worked in 2.7x yet.

Developer note: this issue is caused by a threading conflict between viewport which is catching with its dependency graph in the main thread with the render thread which is modifying main database while it is used by viewport. Not sure how/whether this worked in 2.7x yet.
Member

These crashes existed in 2.79 two when certain things were done. Usually one was able to fix it by rendering from the terminal or locking the UI during rendering. I think these workarounds did not work anymore in Blender 2.8 (haven't tried for quite some time thought).

These crashes existed in 2.79 two when certain things were done. Usually one was able to fix it by rendering from the terminal or locking the UI during rendering. I think these workarounds did not work anymore in Blender 2.8 (haven't tried for quite some time thought).

Added subscriber: @TodorNikolov

Added subscriber: @TodorNikolov
Author

I was testing the recent fix for blender/blender#60136 and noticed that this issue will cause a crash when baking a rigid body cache. Here is a modified version of the .blend file in the original post to reproduce this issue:

bug_script_280_rigid_body.blend

To Reproduce:

  • Open the .blend file
  • Press 'Run Script'

Bake the rigid body world cache

Sometimes the cache bakes successfully without a crash. Repeating step 3 by deleting the cache results in a crash within a few attempts for me.

Error Output:

Error   : EXCEPTION_ACCESS_VIOLATION
Address : 0x00007FF760C36840
Module  : C:\Users\ryanl\Downloads\blender-2.80.0-git.5e626e766459-windows64\blender.exe
I was testing the recent fix for blender/blender#60136 and noticed that this issue will cause a crash when baking a rigid body cache. Here is a modified version of the .blend file in the original post to reproduce this issue: [bug_script_280_rigid_body.blend](https://archive.blender.org/developer/F7102005/bug_script_280_rigid_body.blend) To Reproduce: - Open the .blend file - Press 'Run Script' # Bake the rigid body world cache Sometimes the cache bakes successfully without a crash. Repeating step 3 by deleting the cache results in a crash within a few attempts for me. Error Output: ``` Error : EXCEPTION_ACCESS_VIOLATION Address : 0x00007FF760C36840 Module : C:\Users\ryanl\Downloads\blender-2.80.0-git.5e626e766459-windows64\blender.exe ```

Added subscriber: @FinbarrORiordan

Added subscriber: @FinbarrORiordan

Added subscriber: @ssendam

Added subscriber: @ssendam

Added subscriber: @SamGreen

Added subscriber: @SamGreen

Added subscriber: @georgK

Added subscriber: @georgK

Added subscriber: @Lewiatan

Added subscriber: @Lewiatan

What is the status of this issue? This bug makes using FLIP Fluid plugin impossible because Blender 2.80 Rc 1 crashes during rendering (sometimes as soon as I press "render" (ctrl+F12) ) if FLIP Fluids' objects are present in the scene regardless if I render PNG pictures or a movie.

What is the status of this issue? This bug makes using FLIP Fluid plugin impossible because Blender 2.80 Rc 1 crashes during rendering (sometimes as soon as I press "render" (ctrl+F12) ) if FLIP Fluids' objects are present in the scene regardless if I render PNG pictures or a movie.

Status is: Open, Confirmed, High

This issue is rooting a deep design issues, which were also quite bad in Blender 2.79 but somehow crash was managed to not happen so often.

Possible workaround is to render with Lock Interface option enabled.

Status is: Open, Confirmed, High This issue is rooting a deep design issues, which were also quite bad in Blender 2.79 but somehow crash was managed to not happen so often. Possible workaround is to render with `Lock Interface` option enabled.

Lock Interface did not solve the issue, when I experienced the problem I tried it, without luck, I cannot test it right away, I'm on a hard and tight deadline, but I remember trying everything, even this, and it didn't work

Lock Interface did not solve the issue, when I experienced the problem I tried it, without luck, I cannot test it right away, I'm on a hard and tight deadline, but I remember trying everything, even this, and it didn't work
Author

From FLIP Fluids user reports regarding this render crash issue in Blender 2.80, it seems that the Lock Interface workaround works for roughly half of the users. I have tested systems where locking the interface helps reduce render crashes and some systems where this does not work. In my experience, the render crash frequency increases as the amount of mesh geometry increases.

Render crashes in Blender 2.79 are much less frequent and a workaround of setting the Render Display to Fullscreens seems to work for all FLIP Fluids users.

From FLIP Fluids user reports regarding this render crash issue in Blender 2.80, it seems that the Lock Interface workaround works for roughly half of the users. I have tested systems where locking the interface helps reduce render crashes and some systems where this does not work. In my experience, the render crash frequency increases as the amount of mesh geometry increases. Render crashes in Blender 2.79 are much less frequent and a workaround of setting the Render Display to Fullscreens seems to work for all FLIP Fluids users.

In #60094#723914, @Sergey wrote:
Status is: Open, Confirmed, High

This issue is rooting a deep design issues, which were also quite bad in Blender 2.79 but somehow crash was managed to not happen so often.

Possible workaround is to render with Lock Interface option enabled.

Do you plan to fix this bug soon? It sounds like it will take months to fix it

> In #60094#723914, @Sergey wrote: > Status is: Open, Confirmed, High > > This issue is rooting a deep design issues, which were also quite bad in Blender 2.79 but somehow crash was managed to not happen so often. > > Possible workaround is to render with `Lock Interface` option enabled. Do you plan to fix this bug soon? It sounds like it will take months to fix it

In #60094#724170, @rlguy wrote:
From FLIP Fluids user reports regarding this render crash issue in Blender 2.80, it seems that the Lock Interface workaround works for roughly half of the users. I have tested systems where locking the interface helps reduce render crashes and some systems where this does not work. In my experience, the render crash frequency increases as the amount of mesh geometry increases.

Render crashes in Blender 2.79 are much less frequent and a workaround of setting the Render Display to Fullscreens seems to work for all FLIP Fluids users.

I did notice if I had only FLIP Fluids domain and inflow mesh in the project, Blender 2.80 Rc 1 didn't crash at all. Locking Interface worked for me, I hope it will stay that way.

> In #60094#724170, @rlguy wrote: > From FLIP Fluids user reports regarding this render crash issue in Blender 2.80, it seems that the Lock Interface workaround works for roughly half of the users. I have tested systems where locking the interface helps reduce render crashes and some systems where this does not work. In my experience, the render crash frequency increases as the amount of mesh geometry increases. > > Render crashes in Blender 2.79 are much less frequent and a workaround of setting the Render Display to Fullscreens seems to work for all FLIP Fluids users. I did notice if I had only FLIP Fluids domain and inflow mesh in the project, Blender 2.80 Rc 1 didn't crash at all. Locking Interface worked for me, I hope it will stay that way.

Added subscriber: @Massivetree

Added subscriber: @Massivetree

I really hope this gets fixed soon(and I mean before the stable release of 2.8). Flip Fluids is essentially unusable in Blender 2.8 because of it.

I really hope this gets fixed soon(and I mean before the stable release of 2.8). Flip Fluids is essentially unusable in Blender 2.8 because of it.

This is not only flip fluid related, this could probably affect also elbeem or Mantaflow, since in the end they are generating meshes too

This is not only flip fluid related, this could probably affect also elbeem or Mantaflow, since in the end they are generating meshes too
Member

Added subscribers: @AieKick, @GavinScott

Added subscribers: @AieKick, @GavinScott

Added subscriber: @dr.sybren

Added subscriber: @dr.sybren

Added subscriber: @argelis99

Added subscriber: @argelis99

Is there any work around for this? Even in the official release of 2.8 it crashes on render when using flip fluids. This has been an issue since January, can the Dev team at least tell us if this is something that CAN be fixed? Any communication on such an old high priority issue would be highly appreciated.

Is there any work around for this? Even in the official release of 2.8 it crashes on render when using flip fluids. This has been an issue since January, can the Dev team at least tell us if this is something that CAN be fixed? Any communication on such an old high priority issue would be highly appreciated.

This of course can and will be fixed.

This of course can and will be fixed.

Added subscriber: @Thrilling

Added subscriber: @Thrilling

I'd very much appreciate this issue being fixed. Really excited for 2.8 and Flip Fluids but currently can't render any big files overnight which of course is a big problem.

I'd very much appreciate this issue being fixed. Really excited for 2.8 and Flip Fluids but currently can't render any big files overnight which of course is a big problem.

Added subscriber: @Mitsuma

Added subscriber: @Mitsuma

Added subscriber: @kolibrieirik

Added subscriber: @kolibrieirik

Added subscriber: @jvbandelero

Added subscriber: @jvbandelero

Added subscriber: @PaulAnderson

Added subscriber: @PaulAnderson

I seem to have encountered a work-around. If you go under the Render menu and check Lock Interface it stops the crashes.

I seem to have encountered a work-around. If you go under the Render menu and check Lock Interface it stops the crashes.

In #60094#748133, @PaulAnderson wrote:
I seem to have encountered a work-around. If you go under the Render menu and check Lock Interface it stops the crashes.

Interesting, thank you Alyssa, so far I've been able to render a few dozen frames with no crash where before it would crash after 1 or 2.

> In #60094#748133, @PaulAnderson wrote: > I seem to have encountered a work-around. If you go under the Render menu and check Lock Interface it stops the crashes. Interesting, thank you Alyssa, so far I've been able to render a few dozen frames with no crash where before it would crash after 1 or 2.

In #60094#748133, @PaulAnderson wrote:
I seem to have encountered a work-around. If you go under the Render menu and check Lock Interface it stops the crashes.

Yep, this was a “semi” workaround, Jacques mentioned it here before, but it still crashes after a few frames.

> In #60094#748133, @PaulAnderson wrote: > I seem to have encountered a work-around. If you go under the Render menu and check Lock Interface it stops the crashes. Yep, this was a “semi” workaround, Jacques mentioned it here before, but it still crashes after a few frames.

Added subscriber: @EthanSherriff

Added subscriber: @EthanSherriff

Added subscriber: @LucasVeber

Added subscriber: @LucasVeber

Added subscriber: @LapisSea

Added subscriber: @LapisSea

Added subscriber: @Mrqozy

Added subscriber: @Mrqozy

please fix this!
flip fluids is a must for any water sim in blender, the internal fluid simulation engine is a pain.
its a shame blender devs haven't fixed this bug already.

please fix this! flip fluids is a must for any water sim in blender, the internal fluid simulation engine is a pain. its a shame blender devs haven't fixed this bug already.
Member

Added subscriber: @lichtwerk

Added subscriber: @lichtwerk
Member

@Sergey : Just a note I have added this to 2.81 milestone (since it was set to "High"), I think you are on it already, right?

@Sergey : Just a note I have added this to 2.81 milestone (since it was set to "High"), I think you are on it already, right?

Changed status from 'Open' to: 'Resolved'

Changed status from 'Open' to: 'Resolved'

There are the following changes now made:

Currently it's as stable as we can make it. It is indeed true that interface is better be locked, otherwise render thread might cause conflicts with the viewport (due to python handlers possibly modifying data while it's being drawn by the viewport).

There are ways around it, but needs some design changes. Will be handled outside of the bug tracker as a regular development.

There are the following changes now made: - Dependency graph is always guaranteed to be fully evaluated after handlers for POST depsgraph/frame change (including adding/removing objects, modifying them and so on). - It is possible to access evaluated versions of datablocks from handlers. See https://wiki.blender.org/wiki/Reference/Release_Notes/2.81/Python_API#Handlers Currently it's as stable as we can make it. It is indeed true that interface is better be locked, otherwise render thread might cause conflicts with the viewport (due to python handlers possibly modifying data while it's being drawn by the viewport). There are ways around it, but needs some design changes. Will be handled outside of the bug tracker as a regular development.

Removed subscriber: @kolibrieirik

Removed subscriber: @kolibrieirik

Great news, thanks @Sergey !

Currently it's as stable as we can make it.

Does it mean it may still crashes under certain circumstances?

Great news, thanks @Sergey ! > Currently it's as stable as we can make it. Does it mean it may still crashes under certain circumstances?

Does it mean it may still crashes under certain circumstances?

Yes. If Python handler is adding/removing objects and rendering is happening without Locked Interface.
This was always an issue.

> Does it mean it may still crashes under certain circumstances? Yes. If Python handler is adding/removing objects and rendering is happening without Locked Interface. This was always an issue.

Thank you Sergey. Is the 2.8 build updated after the final release last month with these fixes? Should I reinstall the 2.8 version? Or is this fix only in the 2.81 version?

Thank you Sergey. Is the 2.8 build updated after the final release last month with these fixes? Should I reinstall the 2.8 version? Or is this fix only in the 2.81 version?

The fixes will be available in the net round of nightly builds and in 2.81.

The fixes will be available in the net round of nightly builds and in 2.81.

2.81 release is scheduled for november as far as I remember, I'd bet they won't release a corrective 2.80a version but rather count on 2.81.

2.81 release is scheduled for november as far as I remember, I'd bet they won't release a corrective 2.80a version but rather count on 2.81.

Added subscriber: @MatthewPallotta

Added subscriber: @MatthewPallotta

Added subscriber: @clepsydrae

Added subscriber: @clepsydrae

Added subscriber: @3di

Added subscriber: @3di

still getting the problem here, should I open a new report?

still getting the problem here, should I open a new report?
Member

In #60094#1169192, @3di wrote:
still getting the problem here, should I open a new report?

If RenderLock Interface doesnt solve it for you: yes, please.
https://docs.blender.org/api/master/bpy.app.handlers.html#note-on-altering-data

> In #60094#1169192, @3di wrote: > still getting the problem here, should I open a new report? If `Render` → `Lock Interface` doesnt solve it for you: yes, please. https://docs.blender.org/api/master/bpy.app.handlers.html#note-on-altering-data

Added subscriber: @josiahc

Added subscriber: @josiahc
Sign in to join this conversation.
No Milestone
No project
No Assignees
37 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender-addons#60094
No description provided.