Keyframed parameters are not preserved when frame_change_post handler is used #71908

New Issue

System Information
Operating system: Win 10 Pro
Graphics card: GTX 1080

Blender Version
Broken: v2.81, master, 2018-11-26
Worked: (optional)

EDIT by @EAW

Short description of error

In the .blend file setup, the 'mesh_cache' object would instance a cube over the vertices for display in rendering. An ocean modifier is added to the object with 'Geometry' set to Displace and the 'time' parameter is keyframed over a frame range.

Playback in the viewport shows the ocean modifier evolving shape during the animation. If the animation is rendered, the ocean modifier does not evolve over time and the render result is static.

Here is a script that generates a 2d grid of vertices in the frame_change_post handler:

import bpy

def frame_change_post(scene, depsgraph):
    # Generate a 2d grid of vertices. A cube object is set to be instanced over the vertices for rendering
    vertices = []
    for j in range(100):
        for i in range(100):
            vertices.append((i * 0.05, j * 0.05, 0.0))
    
    mesh_cache_object = bpy.data.objects.get("mesh_cache")
    mesh_cache_object.data.clear_geometry()
    mesh_cache_object.data.from_pydata(vertices, [], [])
    
    depsgraph.update()
    

bpy.app.handlers.frame_change_post.append(frame_change_post)

Exact steps for others to reproduce the error

Here is the .blend file including the script: keyframed_modifier_bug.blend

How to reproduce:

Open .blend file

Press 'Run Script'

Blender > Render > Render Animation

The result is a static animation.

End Edit

Short description of error
I have created a FlipFluids simulation in Blender 2.81 ( 2019-11-25 21:45) and added an Ocean modifier in displacement mode to the fluid_surface. Then I have baked the complete sim from within FlipFluids.
When I move from keyframe to keyframe in the default timeline everything is moving as desired in the default 3D window:
Ocean modifier waves, FlipFluids waves & wake, flags, radars...

When I render the complete animation everything EXCEPT the ocean modifier waves are moving. The ocean modifier waves remain static but the FLIPFLUIDS waves move on that surface.
ALL modifiers and elements ARE enabled for renderings. D-NOISE: AI Denoiser is on (switching denoiser off did not help).
Computer is a HP Z800 with dual Xeons, 72 GB of vRAM and a GTX 1060.

If I render & save the keyframes manually individually by hand the ocean surface moves! Which means I get the proper result.

As its working when rendering&saving each frame manually as image (Png) and its not giving the same result when saving out the same images (Png) when I hit render animation, I do suppose this might be a bug in 2.81.
Running indiviual frames via Renderpilot did not yield the proper result either. So I do suspect it might be that there is not enough time gap between saving the last frame and rendering the next. Or, something does not get purged properly.

For propriatary reasons I can not share the blend file.

**System Information** Operating system: Win 10 Pro Graphics card: GTX 1080 **Blender Version** Broken: v2.81, master, 2018-11-26 Worked: (optional) --- **EDIT** by @EAW **Short description of error** > In the .blend file setup, the 'mesh_cache' object would instance a cube over the vertices for display in rendering. An ocean modifier is added to the object with 'Geometry' set to Displace and the 'time' parameter is keyframed over a frame range. > > Playback in the viewport shows the ocean modifier evolving shape during the animation. If the animation is rendered, the ocean modifier does not evolve over time and the render result is static. > > Here is a script that generates a 2d grid of vertices in the frame_change_post handler: > > ``` > import bpy > > def frame_change_post(scene, depsgraph): > # Generate a 2d grid of vertices. A cube object is set to be instanced over the vertices for rendering > vertices = [] > for j in range(100): > for i in range(100): > vertices.append((i * 0.05, j * 0.05, 0.0)) > > mesh_cache_object = bpy.data.objects.get("mesh_cache") > mesh_cache_object.data.clear_geometry() > mesh_cache_object.data.from_pydata(vertices, [], []) > > depsgraph.update() > > > bpy.app.handlers.frame_change_post.append(frame_change_post) > ``` > **Exact steps for others to reproduce the error** > Here is the .blend file including the script: [keyframed_modifier_bug.blend](https://archive.blender.org/developer/F8175418/keyframed_modifier_bug.blend) > > How to reproduce: > 1. Open .blend file > 2. Press 'Run Script' > 3. Blender > Render > Render Animation > > The result is a static animation. **End Edit** --- **Short description of error** I have created a FlipFluids simulation in Blender 2.81 ( 2019-11-25 21:45) and added an Ocean modifier in displacement mode to the fluid_surface. Then I have baked the complete sim from within FlipFluids. When I move from keyframe to keyframe in the default timeline everything is moving as desired in the default 3D window: Ocean modifier waves, FlipFluids waves & wake, flags, radars... When I render the complete animation everything EXCEPT the ocean modifier waves are moving. The ocean modifier waves remain static but the FLIPFLUIDS waves move on that surface. ALL modifiers and elements ARE enabled for renderings. D-NOISE: AI Denoiser is on (switching denoiser off did not help). Computer is a HP Z800 with dual Xeons, 72 GB of vRAM and a GTX 1060. If I render & save the keyframes manually individually by hand the ocean surface moves! Which means I get the proper result. As its working when rendering&saving each frame manually as image (Png) and its not giving the same result when saving out the same images (Png) when I hit render animation, I do suppose this might be a bug in 2.81. Running indiviual frames via Renderpilot did not yield the proper result either. So I do suspect it might be that there is not enough time gap between saving the last frame and rendering the next. Or, something does not get purged properly. For propriatary reasons I can not share the blend file.

Added subscriber: @Prettypicturegirl

#78866 was marked as duplicate of this issue

#77008 was marked as duplicate of this issue

#73686 was marked as duplicate of this issue

#75626 was marked as duplicate of this issue

Attached two samples showing the different results.

SingleFrames.rar

AnimationFrames.rar

Attached two samples showing the different results. [SingleFrames.rar](https://archive.blender.org/developer/F8168123/SingleFrames.rar) [AnimationFrames.rar](https://archive.blender.org/developer/F8168122/AnimationFrames.rar)

Added subscriber: @rlguy

I am able to reproduce this issue in the FLIP Fluids addon. I have created a minimal script and .blend file that can reproduce the issue. It seems that modifiers with parameters that are keyframed do not update when the modifier is on a mesh that is procedurally generated in the frame_change_post handler.

Here is a script that generates a 2d grid of vertices in the frame_change_post handler:

import bpy

def frame_change_post(scene, depsgraph):
    # Generate a 2d grid of vertices. A cube object is set to be instanced over the vertices for rendering
    vertices = []
    for j in range(100):
        for i in range(100):
            vertices.append((i * 0.05, j * 0.05, 0.0))
    
    mesh_cache_object = bpy.data.objects.get("mesh_cache")
    mesh_cache_object.data.clear_geometry()
    mesh_cache_object.data.from_pydata(vertices, [], [])
    
    depsgraph.update()
    

bpy.app.handlers.frame_change_post.append(frame_change_post)

In the .blend file setup, the 'mesh_cache' object would instance a cube over the vertices for display in rendering. An ocean modifier is added to the object with 'Geometry' set to Displace and the 'time' parameter is keyframed over a frame range.

Playback in the viewport shows the ocean modifier evolving shape during the animation. If the animation is rendered, the ocean modifier does not evolve over time and the render result is static.

Here is the .blend file including the script: keyframed_modifier_bug.blend

How to reproduce:

Open .blend file
Press 'Run Script'
Blender > Render > Render Animation

The result is a static animation.

I am able to reproduce this issue in the FLIP Fluids addon. I have created a minimal script and .blend file that can reproduce the issue. It seems that modifiers with parameters that are keyframed do not update when the modifier is on a mesh that is procedurally generated in the frame_change_post handler. Here is a script that generates a 2d grid of vertices in the frame_change_post handler: ``` import bpy def frame_change_post(scene, depsgraph): # Generate a 2d grid of vertices. A cube object is set to be instanced over the vertices for rendering vertices = [] for j in range(100): for i in range(100): vertices.append((i * 0.05, j * 0.05, 0.0)) mesh_cache_object = bpy.data.objects.get("mesh_cache") mesh_cache_object.data.clear_geometry() mesh_cache_object.data.from_pydata(vertices, [], []) depsgraph.update() bpy.app.handlers.frame_change_post.append(frame_change_post) ``` In the .blend file setup, the 'mesh_cache' object would instance a cube over the vertices for display in rendering. An ocean modifier is added to the object with 'Geometry' set to Displace and the 'time' parameter is keyframed over a frame range. Playback in the viewport shows the ocean modifier evolving shape during the animation. If the animation is rendered, the ocean modifier does not evolve over time and the render result is static. Here is the .blend file including the script: [keyframed_modifier_bug.blend](https://archive.blender.org/developer/F8175414/keyframed_modifier_bug.blend) How to reproduce: 1. Open .blend file 2. Press 'Run Script' 3. Blender > Render > Render Animation The result is a static animation.

Added subscriber: @mano-wii

Please follow our submission template and guidelines, also read these tips about bug reports, and make a complete, valid bug report, with required info, precise description of the issue, precise steps to reproduce it, small and simple .blend and/or other files to do so if needed, etc.
Also explained here: https://www.youtube.com/watch?v=JTD0OJq_rF4

Also keep in mind that Flip Fluid Addon is not maintained by blender.
So if the bug is related to it please report the issue to the original author instead.

Please follow our [submission template and guidelines](https:*developer.blender.org/maniphest/task/edit/form/1/), also read [these tips about bug reports](https:*wiki.blender.org/wiki/Process/Bug_Reports), and make a complete, valid bug report, with required info, precise description of the issue, precise steps to reproduce it, small and simple .blend and/or other files to do so if needed, etc. Also explained here: https://www.youtube.com/watch?v=JTD0OJq_rF4 Also keep in mind that Flip Fluid Addon is not maintained by blender. So if the bug is related to it please report the issue to the original author instead.

Added subscriber: @Germano

@Germano

I do not understand what your issue is.

I am not a developer and tried my very best to give all information necessary in the required format.

Further the FlipFluids add-on developer himself RLGuy answered in a reply here and confirmed and even showed how the bug appears. It's clearly a bug in Blender and not in the add-on.

So, why are you downvoting a user reported bug instead of clearly indicating what further info you need?
That's not at all serving the product quality....

@Germano I do not understand what your issue is. I am not a developer and tried my very best to give all information necessary in the required format. Further the FlipFluids add-on developer himself RLGuy answered in a reply here and confirmed and even showed how the bug appears. It's clearly a bug in Blender and not in the add-on. So, why are you downvoting a user reported bug instead of clearly indicating what further info you need? That's not at all serving the product quality....

Further rlguy gave you a suitable blend file and detailed information of how to reproduce the issue.
Please be so kind and read his reply. It's very detailed.

I am not a Blender specialist and I can not share such a blend file as I have not enough know how of how to create such a small file.
Neither can I provide the huge original blend file as it has copyrighted design geometry within.

Further rlguy gave you a suitable blend file and detailed information of how to reproduce the issue. Please be so kind and read his reply. It's very detailed. I am not a Blender specialist and I can not share such a blend file as I have not enough know how of how to create such a small file. Neither can I provide the huge original blend file as it has copyrighted design geometry within.

In the submission template and guidelines there is a clear distinction between:
Short description of error
and
Exact steps for others to reproduce the error

The steps to reproduce the problem need to be simple, without the need to install external addons.

In the [submission template and guidelines](https://developer.blender.org/maniphest/task/edit/form/1/) there is a clear distinction between: **Short description of error** and **Exact steps for others to reproduce the error** The steps to reproduce the problem need to be simple, without the need to install external addons.

Added subscribers: @fran-12, @EAW

@fran-12

Update the title of this report to be “Keyframed modifier parameters do not update on a mesh that is procedurally generated in the frame_change_post handler.”
It is a much more accurate description of the issue. I would update it for you but I am not a moderator.

Then copy and paste @rlguy’s comment above you original description to make this a valid bug report.

Under short description of error paste “Playback in the viewport shows the ocean modifier evolving shape during the animation. If the animation is rendered, the ocean modifier does not evolve over time and the render result is static.”

Paste the rest of his comment under “Exact steps for others to reproduce the error”.

This will make the bug report follow the template.

I am unable to test his method myself today as I forgot to pack my laptop cord and the stores are closed due to a national holiday in the USA.

@fran-12 Update the title of this report to be “Keyframed modifier parameters do not update on a mesh that is procedurally generated in the frame_change_post handler.” It is a much more accurate description of the issue. I would update it for you but I am not a moderator. Then copy and paste @rlguy’s comment above you original description to make this a valid bug report. Under short description of error paste “Playback in the viewport shows the ocean modifier evolving shape during the animation. If the animation is rendered, the ocean modifier does not evolve over time and the render result is static.” Paste the rest of his comment under “Exact steps for others to reproduce the error”. This will make the bug report follow the template. I am unable to test his method myself today as I forgot to pack my laptop cord and the stores are closed due to a national holiday in the USA.

Guess than I am not in the position to help you to fix a clearly existing bug in the current release. Im to much à newbie to describe it in detail.

Rlguy provided exactly what you asked for in his reply, but you seem not content with this either.

I'm fairly new to Blender and was hoping to be able to contribute to getting rid of a bug.
As this seems a complex thing to do, guess I will not endeavor in the future to file bug reports.

Which is ashame because it would have actually improved Blender...

Guess than I am not in the position to help you to fix a clearly existing bug in the current release. Im to much à newbie to describe it in detail. Rlguy provided exactly what you asked for in his reply, but you seem not content with this either. I'm fairly new to Blender and was hoping to be able to contribute to getting rid of a bug. As this seems a complex thing to do, guess I will not endeavor in the future to file bug reports. Which is ashame because it would have actually improved Blender...

Added subscriber: @lichtwerk

@rlguy: could you check your file again?
For me, the is no script in ocean_displacement_test.blend, also no "mesh_cache" object...

@rlguy: could you check your file again? For me, the is no script in `ocean_displacement_test.blend`, also no "mesh_cache" object...

Thanks Evan,

very kind of you. I am traveling right now, and can do this tomorrow.
Should I file a new bug report with the format you described? Or can I shuffle things around in this bug report?

Thanks,

Fran

Thanks Evan, very kind of you. I am traveling right now, and can do this tomorrow. Should I file a new bug report with the format you described? Or can I shuffle things around in this bug report? Thanks, Fran

@Prettypicturegirl : we can do this, but waiting for @rlguy to provide the updated file.
We'll get there :)
[But of course it helps if everything is there from the very start -- going back and forth just takes time, and time is really precious with the ammount of work to be done in the tracker atm...]
Everything good everyone?

@Prettypicturegirl : we can do this, but waiting for @rlguy to provide the updated file. We'll get there :) [But of course it helps if everything is there from the very start -- going back and forth just takes time, and time is really precious with the ammount of work to be done in the tracker atm...] Everything good everyone?

Thank you Philipp. Very kind of you. And yes I'm happy now, especially if you can fix it.
I just recently switched from an expensive commercial copy of V-Ray for Rhino to Blender and am really excited what and how much more can be done with it.

Thank you Philipp. Very kind of you. And yes I'm happy now, especially if you can fix it. I just recently switched from an expensive commercial copy of V-Ray for Rhino to Blender and am really excited what and how much more can be done with it.

In #71908#820445, @lichtwerk wrote:
@rlguy: could you check your file again?
For me, the is no script in ocean_displacement_test.blend, also no "mesh_cache" object...

My apologies, I had uploaded the incorrect file. I have updated the comment with the correct file: keyframed_modifier_bug.blend

> In #71908#820445, @lichtwerk wrote: > @rlguy: could you check your file again? > For me, the is no script in `ocean_displacement_test.blend`, also no "mesh_cache" object... My apologies, I had uploaded the incorrect file. I have updated the comment with the correct file: [keyframed_modifier_bug.blend](https://archive.blender.org/developer/F8175418/keyframed_modifier_bug.blend)

Added subscriber: @Sergey

@rlguy
I am not a programmer, so pardon me if this is naive, but should it be

def frame_change_post_handler(scene, depsgraph):

Instead of

def frame_change_post(scene, depsgraph):

See

https://wiki.blender.org/wiki/Reference/Release_Notes/2.81/Python_API#Handlers

And

In blender/blender-addons#60094#773490, @Sergey wrote:
There are the following changes now made:

Dependency graph is always guaranteed to be fully evaluated after handlers for POST depsgraph/frame change (including adding/removing objects, modifying them and so on).

It is possible to access evaluated versions of datablocks from handlers. See https://wiki.blender.org/wiki/Reference/Release_Notes/2.81/Python_API#Handlers

Currently it's as stable as we can make it. It is indeed true that interface is better be locked, otherwise render thread might cause conflicts with the viewport (due to python handlers possibly modifying data while it's being drawn by the viewport).

There are ways around it, but needs some design changes. Will be handled outside of the bug tracker as a regular development.

Really wish my laptop battery wasn’t right now :)

@

@rlguy I am not a programmer, so pardon me if this is naive, but should it be def frame_change_post_handler(scene, depsgraph): Instead of def frame_change_post(scene, depsgraph): See https://wiki.blender.org/wiki/Reference/Release_Notes/2.81/Python_API#Handlers And > In blender/blender-addons#60094#773490, @Sergey wrote: > There are the following changes now made: > > - Dependency graph is always guaranteed to be fully evaluated after handlers for POST depsgraph/frame change (including adding/removing objects, modifying them and so on). > > - It is possible to access evaluated versions of datablocks from handlers. See https://wiki.blender.org/wiki/Reference/Release_Notes/2.81/Python_API#Handlers > > Currently it's as stable as we can make it. It is indeed true that interface is better be locked, otherwise render thread might cause conflicts with the viewport (due to python handlers possibly modifying data while it's being drawn by the viewport). > > There are ways around it, but needs some design changes. Will be handled outside of the bug tracker as a regular development. Really wish my laptop battery wasn’t right now :) @

In #71908#820446, @Prettypicturegirl wrote:
Thanks Evan,

very kind of you. I am traveling right now, and can do this tomorrow.
Should I file a new bug report with the format you described? Or can I shuffle things around in this bug report?

Thanks,

Fran

You are most welcome!

> In #71908#820446, @Prettypicturegirl wrote: > Thanks Evan, > > very kind of you. I am traveling right now, and can do this tomorrow. > Should I file a new bug report with the format you described? Or can I shuffle things around in this bug report? > > Thanks, > > Fran You are most welcome!

@EAW The name/identifier of the handler function can be set to any custom name (under the constraints of what is valid in Python). The handler function just needs to be appended to the bpy.app.handlers.frame_change_post list so that Blender runs the function on a frame change.

For example, this should also be valid API usage:

import bpy

def my_custom_handler(scene, depsgraph):
    pass
    

bpy.app.handlers.frame_change_post.append(my_custom_handler)

@EAW The name/identifier of the handler function can be set to any custom name (under the constraints of what is valid in Python). The handler function just needs to be appended to the *bpy.app.handlers.frame_change_post* list so that Blender runs the function on a frame change. For example, this should also be valid API usage: ``` import bpy def my_custom_handler(scene, depsgraph): pass bpy.app.handlers.frame_change_post.append(my_custom_handler) ```

Thank you for letting me know @rlguy

@Prettypicturegirl I have edited the title and description for you.

Thank you for letting me know @rlguy @Prettypicturegirl I have edited the title and description for you.

Perfect, see I'd never had the chance to rephrase tjis way, as I lack the know how in Blender at the moment. Thank you Evan.

This issue was referenced by befc082736

This issue was referenced by befc0827362da04cfff41e057e789159ab5bb0b5

Changed status from 'Confirmed' to: 'Resolved'

Changed status from 'Resolved' to: 'Confirmed'

Re-opening.
The fix caused #73029. Need to re-iterate over solution.

Re-opening. The fix caused #73029. Need to re-iterate over solution.

Added subscriber: @dr.sybren

#75626 (Changing properties of an object from a frame_change_post handler blocks animation) describes the same issue, albeit in a simpler way, with just three animated balls. The blend file & reproduction steps might be nicer to use for debugging purposes.

Developer note.

The system, works in the following way: when frame_change_post modifies some datablock an extra update of the dependency graph is done, but this update will not do animation update (with an intent to allow to manually apply changes on top of animation).

For the viewport the flow in this usecase goes as following:

First pass of update in BKE_scene_graph_update_for_newframe is done for dependency graph + animation
This pass will evaluation animation and, since this is an active viewport dependency graph, the animated values are copied from evaluated IDs to original ones (which allows all tools to work nicely, without modifying every one of them).
frame_change_post modifies some datablock (NOTE: modification happens on original datablock, with an intent that system will pick changes up and propagate to all related dependency graphs).
This is detected in BKE_scene_graph_update_for_newframe and second round of updates is performed. This time without evaluating manimation.
The evaluated datablock corresponding to the modified one is updated from the updated one. And since at step 2 animation was copied from evaluated to original datablock everything appears to work correct.

For the non-viewport depsgraph the difference is at step 2: there is no copy of new animated property value from evaluated to original. This makes step 5 to "override" evaluated animated value with the one from the viewport, which is likely to be wrong (since viewport is evaluated at different frame than the non-viewport depsgraph).

Since we don't have any way to know which properties were modified, I am not sure what could be a solution without breaking API, or without breaking existing mindset, or without loosing ability to override animated values from the handler.

P.S. Since I didn't have time to look into for so long, unassigning so then other developers are more encouraged to pick up and help.

Developer note. The system, works in the following way: when `frame_change_post` modifies some datablock an extra update of the dependency graph is done, but this update will not do animation update (with an intent to allow to manually apply changes on top of animation). For the viewport the flow in this usecase goes as following: 1. First pass of update in `BKE_scene_graph_update_for_newframe` is done for dependency graph + animation 2. This pass will evaluation animation and, since this is an active viewport dependency graph, the animated values are copied from evaluated IDs to original ones (which allows all tools to work nicely, without modifying every one of them). 3. `frame_change_post` modifies some datablock (NOTE: modification happens on original datablock, with an intent that system will pick changes up and propagate to all related dependency graphs). 4. This is detected in `BKE_scene_graph_update_for_newframe` and second round of updates is performed. This time without evaluating manimation. 5. The evaluated datablock corresponding to the modified one is updated from the updated one. And since at step 2 animation was copied from evaluated to original datablock everything appears to work correct. For the non-viewport depsgraph the difference is at step 2: there is no copy of new animated property value from evaluated to original. This makes step 5 to "override" evaluated animated value with the one from the viewport, which is likely to be wrong (since viewport is evaluated at different frame than the non-viewport depsgraph). Since we don't have any way to know which properties were modified, I am not sure what could be a solution without breaking API, or without breaking existing mindset, or without loosing ability to override animated values from the handler. P.S. Since I didn't have time to look into for so long, unassigning so then other developers are more encouraged to pick up and help.

Added subscribers: @StephenHamacek, @ZedDB, @StanislavOvcharov, @Alex_Rom, @jpbouza-4, @iss

For the non-viewport depsgraph the difference is at step 2: there is no copy of new animated property value from evaluated to original.

@Sergey how would you feel about a way to, instead of updating the original datablock from the handler, update the evaluated datablock? To me this feels more in line with "updating after the animation has run". To avoid breaking the existing API, this could be done from a separate callback, like frame_change_post_animation. The callback could get the evaluated scene, and thus iterate over its evaluated objects.

> For the non-viewport depsgraph the difference is at step 2: there is no copy of new animated property value from evaluated to original. @Sergey how would you feel about a way to, instead of updating the original datablock from the handler, update the evaluated datablock? To me this feels more in line with "updating after the animation has run". To avoid breaking the existing API, this could be done from a separate callback, like `frame_change_post_animation`. The callback could get the evaluated scene, and thus iterate over its evaluated objects.

How would you feel about a way to, instead of updating the original datablock from the handler, update the evaluated datablock?

While this sounds like an easy solution in this specific case, it opens up a huge amount of other issues and inconsistencies:

Generally, no one should ever edit evaluated datablock. This is what developers are to follow in both C and Python code.
Currently handlers are acting like a global modification. This means that from a handler you can modify (i.e. procedurally generate) your scene, which will be the same in all the "contexts" (viewport, another window, F12 render).
Modifications of a datablock are not necessary limited to just overriding scalar properties. You might also want to override ID pointer, which could involve rebuilding relations and pulling new datablock to the graph. Surely, you might get this thing to work, but the next point comes in place.
You might want to create new datablocks and use them for overriding various things. So your suggestion escalates to support of entirely separate bMain database for the evaluated scene.

This list can go on, with every item going deeper and deeper in design reconsiderations in all areas. So while this is not an invalid plausible path, it needs careful thoughts and clear design put on paper rather than be accepted on a feelings that it solves this specific case.

To avoid breaking the existing API, this could be done from a separate callback, like frame_change_post_animation. The callback could get the evaluated scene, and thus iterate over its evaluated objects.

Please have a look at the existing handler signature. You can access both original and evaluated datablocks since you do have access to the dependency graph. Accessing evaluated objects is not the problem here from the API point of view. What affects on API is If we say that handlers are to modify evaluated datablocks, we are likely to break all existing handlers. One would argue that existing behavior can be preserved, but it will be same limited, it might very quickly get in a way, and having 2 semi-working ways of doing something just adds even more confusion.

> How would you feel about a way to, instead of updating the original datablock from the handler, update the evaluated datablock? While this sounds like an easy solution in this specific case, it opens up a huge amount of other issues and inconsistencies: - Generally, no one should ever edit evaluated datablock. This is what developers are to follow in both C and Python code. - Currently handlers are acting like a global modification. This means that from a handler you can modify (i.e. procedurally generate) your scene, which will be the same in all the "contexts" (viewport, another window, F12 render). - Modifications of a datablock are not necessary limited to just overriding scalar properties. You might also want to override ID pointer, which could involve rebuilding relations and pulling new datablock to the graph. Surely, you might get this thing to work, but the next point comes in place. - You might want to create new datablocks and use them for overriding various things. So your suggestion escalates to support of entirely separate bMain database for the evaluated scene. This list can go on, with every item going deeper and deeper in design reconsiderations in all areas. So while this is not an invalid plausible path, it needs careful thoughts and clear design put on paper rather than be accepted on a feelings that it solves this specific case. > To avoid breaking the existing API, this could be done from a separate callback, like `frame_change_post_animation`. The callback could get the evaluated scene, and thus iterate over its evaluated objects. Please have a look at the existing handler signature. You can access both original and evaluated datablocks since you do have access to the dependency graph. Accessing evaluated objects is not the problem here from the API point of view. What affects on API is **If** we say that handlers are to modify evaluated datablocks, we are likely to break all existing handlers. One would argue that existing behavior can be preserved, but it will be same limited, it might very quickly get in a way, and having 2 semi-working ways of doing something just adds even more confusion.

Thanks for the clear explanation @sergey.

I've been thinking about this, and any solution I see keeps boiling down to keeping track of which properties are changed by frame_change_post handlers. That way we can avoid doing a full copy (original → evaluated) in step 5 above and only copy those properties that were updated in the frame_change_post handler. That would keep animated property values intact, and would also allow overriding their values from Python.

Alternatively, it could be possible to make the animation system run but disallow overwriting properties that were changed by frame_change_post. However, this would put a burden on the animation evaluation to check each and every property set, so that'll be bad for performance for sure. And in the end it also boils down to "keep track of which properties were changed".

Keeping track of which properties were changed could be automatically done by RNA, but I suspect that adding even a single boolean check on all RNA setters will negatively impact performance outside of the scope of frame change handlers. Alternatively this could be done explicitly, as something special that's required for frame_change_post handler functions; something like datablock.update_tag_attribute('some_attribute'). I'm also aprehensive of this solution, as it's easy to forget that this is a necessary step, and I had rather Blender do this automatically. However, I would prefer explicitly tagging attributes over worse overall performance.

This approach will not gracefully handle those cases where a single value in DNA is exposed as multiple values in RNA (for example Object.hide_viewport and hide_render)

Modifications of a datablock are not necessary limited to just overriding scalar properties. You might also want to override ID pointer, which could involve rebuilding relations and pulling new datablock to the graph.

Good point. When new datablocks are pulled into the graph, I'm guessing they won't get animated property values now either?

Thanks for the clear explanation @sergey. I've been thinking about this, and any solution I see keeps boiling down to keeping track of which properties are changed by `frame_change_post` handlers. That way we can avoid doing a full copy (original → evaluated) in [step 5 above](https://developer.blender.org/T71908#916261) and only copy those properties that were updated in the `frame_change_post` handler. That would keep animated property values intact, and would also allow overriding their values from Python. Alternatively, it could be possible to make the animation system run but disallow overwriting properties that were changed by `frame_change_post`. However, this would put a burden on the animation evaluation to check each and every property set, so that'll be bad for performance for sure. And in the end it also boils down to "keep track of which properties were changed". Keeping track of which properties were changed could be automatically done by RNA, but I suspect that adding even a single boolean check on all RNA setters will negatively impact performance outside of the scope of frame change handlers. Alternatively this could be done explicitly, as something special that's required for `frame_change_post` handler functions; something like `datablock.update_tag_attribute('some_attribute')`. I'm also aprehensive of this solution, as it's easy to forget that this is a necessary step, and I had rather Blender do this automatically. However, I would prefer explicitly tagging attributes over worse overall performance. This approach will not gracefully handle those cases where a single value in DNA is exposed as multiple values in RNA (for example `Object.hide_viewport` and `hide_render`) > Modifications of a datablock are not necessary limited to just overriding scalar properties. You might also want to override ID pointer, which could involve rebuilding relations and pulling new datablock to the graph. Good point. When new datablocks are pulled into the graph, I'm guessing they won't get animated property values now either?

I've been thinking about this, and any solution I see keeps boiling down to keeping track of which properties are changed by frame_change_post handlers. That way we can avoid doing a full copy (original → evaluated) in step 5 above and only copy those properties that were updated in the frame_change_post handler. That would keep animated property values intact, and would also allow overriding their values from Python.

It indeed sounds more like we do need storage of some sort, but it's.. Complicated.

Complication #1: due to the nature of RNA and python scripting you would need to inform all dependency graphs about all property changes. Imagine having importer implemented in Python, ugh :(

Complication #2: Copying becomes more complicated, it needs to know that at some point it need to do one thing, and at another point another thing.

Complication #3: Handlers do not only modify properties via RNA, they might run C function (via bpy utility or via operator) which will perform modification, without using RNA API.

When new datablocks are pulled into the graph, I'm guessing they won't get animated property values now either?

If the handler adds new objects with animation data, yeah, probably it wouldn't. get animation evaluated (since second round of updates only updates tags, without time).

> I've been thinking about this, and any solution I see keeps boiling down to keeping track of which properties are changed by frame_change_post handlers. That way we can avoid doing a full copy (original → evaluated) in step 5 above and only copy those properties that were updated in the frame_change_post handler. That would keep animated property values intact, and would also allow overriding their values from Python. It indeed sounds more like we do need storage of some sort, but it's.. Complicated. Complication #1: due to the nature of RNA and python scripting you would need to inform all dependency graphs about all property changes. Imagine having importer implemented in Python, ugh :( Complication #2: Copying becomes more complicated, it needs to know that at some point it need to do one thing, and at another point another thing. Complication #3: Handlers do not only modify properties via RNA, they might run C function (via bpy utility or via operator) which will perform modification, without using RNA API. > When new datablocks are pulled into the graph, I'm guessing they won't get animated property values now either? If the handler adds new objects with animation data, yeah, probably it wouldn't. get animation evaluated (since second round of updates only updates tags, without time).

In #71908#928864, @Sergey wrote:
Complication #1: due to the nature of RNA and python scripting you would need to inform all dependency graphs about all property changes. Imagine having importer implemented in Python, ugh :(

That's only an issue when you want to have existing animation also applied. So the importer would have to be a rather special one, importing some animated data (otherwise it wouldn't have to be run on frame change) but still allowing other animations to be applied as well. I don't know of any importers that do this. Could provide a nice workflow of hand-animating some properties while others are animated by the importer.

Complication #2: Copying becomes more complicated, it needs to know that at some point it need to do one thing, and at another point another thing.

That's only if we want to have this system work everywhere, all the time. From what I can see, it's only an issue in the frame-change-post handler, so applying the changed properties onto the already-evaluated copies could be done in a specialised function. That way simple copies remain simple.

Complication #3: Handlers do not only modify properties via RNA, they might run C function (via bpy utility or via operator) which will perform modification, without using RNA API.

Hmmm yes. We could consider this unsupported (as it is unsupported now as well), and only allow direct RNA prop modifications in a frame-change-post handler.

A possible workaround could be to inspect the C code and figure out which RNA properties it changes, and tagging those for update from the handler in Python. I'm not too thrilled about that approach, though, as it's not friendly and would create unclean code where the C and Python code are tightly coupled. I see this break easily.

> In #71908#928864, @Sergey wrote: > Complication #1: due to the nature of RNA and python scripting you would need to inform all dependency graphs about all property changes. Imagine having importer implemented in Python, ugh :( That's only an issue when you want to have existing animation also applied. So the importer would have to be a rather special one, importing some animated data (otherwise it wouldn't have to be run on frame change) but still allowing other animations to be applied as well. I don't know of any importers that do this. Could provide a nice workflow of hand-animating some properties while others are animated by the importer. > Complication #2: Copying becomes more complicated, it needs to know that at some point it need to do one thing, and at another point another thing. That's only if we want to have this system work everywhere, all the time. From what I can see, it's only an issue in the frame-change-post handler, so applying the changed properties onto the already-evaluated copies could be done in a specialised function. That way simple copies remain simple. > Complication #3: Handlers do not only modify properties via RNA, they might run C function (via bpy utility or via operator) which will perform modification, without using RNA API. Hmmm yes. We could consider this unsupported (as it is unsupported now as well), and only allow direct RNA prop modifications in a frame-change-post handler. A possible workaround could be to inspect the C code and figure out which RNA properties it changes, and tagging those for update from the handler in Python. I'm not too thrilled about that approach, though, as it's not friendly and would create unclean code where the C and Python code are tightly coupled. I see this break easily.

That's only an issue when you want to have existing animation also applied. So the importer would have to be a rather special one, importing some animated data (otherwise it wouldn't have to be run on frame change) but still allowing other animations to be applied as well. I don't know of any importers that do this.

You don't want to check whether property is animated or not prior to dependency graph tag. While tagging is not fully cheap, it is way cheaper than checking whether property is animated or not.

Could provide a nice workflow of hand-animating some properties while others are animated by the importer.

Nice workflow would be to "integrate" such properties hand-animation as a part of datablock evaluation. I do not see a scene-global handler which traverses objects a nice workflow.

Such things are not importer-specific, so not sure how this is relevant.

That's only if we want to have this system work everywhere, all the time. From what I can see, it's only an issue in the frame-change-post handler, so applying the changed properties onto the already-evaluated copies could be done in a specialised function. That way simple copies remain simple.

It needs to be a generic enough solution, which will live within design of many areas. You can not be limiting considerations to just "properties change of frame-change-post handler".

Hmmm yes. We could consider this unsupported (as it is unsupported now as well), and only allow direct RNA prop modifications in a frame-change-post handler.

This isn't how the handler is used.

A possible workaround could be to inspect the C code and figure out which RNA properties it changes, and tagging those for update from the handler in Python.

You can't think in terms of RNA properties.
One of usages is to generated meshes, for example, where handler can remove vertices, add new ones.

Long story short: what do I see here so far is an attempts to address some individual aspects of the problem. What I do not see here is a systematic approach which will analyze how functionality is used, how it is expected to behave, how can solution realistically fit existing design of all related areas.

> That's only an issue when you want to have existing animation also applied. So the importer would have to be a rather special one, importing some animated data (otherwise it wouldn't have to be run on frame change) but still allowing other animations to be applied as well. I don't know of any importers that do this. You don't want to check whether property is animated or not prior to dependency graph tag. While tagging is not fully cheap, it is way cheaper than checking whether property is animated or not. > Could provide a nice workflow of hand-animating some properties while others are animated by the importer. Nice workflow would be to "integrate" such properties hand-animation as a part of datablock evaluation. I do not see a scene-global handler which traverses objects a nice workflow. Such things are not importer-specific, so not sure how this is relevant. > That's only if we want to have this system work everywhere, all the time. From what I can see, it's only an issue in the frame-change-post handler, so applying the changed properties onto the already-evaluated copies could be done in a specialised function. That way simple copies remain simple. It needs to be a generic enough solution, which will live within design of many areas. You can not be limiting considerations to just "properties change of frame-change-post handler". > Hmmm yes. We could consider this unsupported (as it is unsupported now as well), and only allow direct RNA prop modifications in a frame-change-post handler. This isn't how the handler is used. > A possible workaround could be to inspect the C code and figure out which RNA properties it changes, and tagging those for update from the handler in Python. You can't think in terms of RNA properties. One of usages is to generated meshes, for example, where handler can remove vertices, add new ones. ---- Long story short: what do I see here so far is an attempts to address some individual aspects of the problem. What I do not see here is a systematic approach which will analyze how functionality is used, how it is expected to behave, how can solution realistically fit existing design of all related areas.

After discussion with @Sergey we arrived at the following. From what we can see, the following are reasons for having a frame_change_post handler:

Reading datablocks. This is doesn't involve any modification, so is not interesting in the scope of this task.
Writing to existing datablocks, to either apply a change on top of the original data, or to fully define the state of the datablock.
Writing to existing datablocks, to apply a change to the evaluated data (so on top of animation, simulations etc.).
Creating new datablocks.

Case 2. is already covered by Blender's current behaviour.

Case 3. could be seen as the frame-change handler being part of the depsgraph evaluation. This could be implemented via customisable Python nodes in the depsgraph. For performance reasons, it's probably a better idea to not do this, and to allow modifying evaluated datablocks from one single function: the frame-change-post handler. However, these changes should not modify (or require modification of) existing relations for sanity's sake. This is likely going to work fine with the current RNA update callbacks. They expect a pointer to original data, and will tag the data for update, but effectively they'll be tagging the evaluated datablock for update. These are not iterated over by the depsgraph, so tagging won't have any impact. For performance & clarity we can explicitly let the tagging functions ignore evaluated datablocks.

Case 4. requires some extra care. We could resolve this by only allowing the creation of new ID datablocks (most importantly Objects) from the frame-change-pre handler. This handler is called after the scene's current frame has been updated, but before the dependency graph evaluation of the new frame.

After discussion with @Sergey we arrived at the following. From what we can see, the following are reasons for having a `frame_change_post` handler: 1. Reading datablocks. This is doesn't involve any modification, so is not interesting in the scope of this task. 2. Writing to existing datablocks, to either apply a change on top of the original data, or to fully define the state of the datablock. 3. Writing to existing datablocks, to apply a change to the evaluated data (so on top of animation, simulations etc.). 4. Creating new datablocks. Case 2. is already covered by Blender's current behaviour. Case 3. could be seen as the frame-change handler being part of the depsgraph evaluation. This could be implemented via customisable Python nodes in the depsgraph. For performance reasons, it's probably a better idea to not do this, and to allow modifying evaluated datablocks from one single function: the frame-change-post handler. However, these changes should not modify (or require modification of) existing relations for sanity's sake. This is likely going to work fine with the current RNA update callbacks. They expect a pointer to original data, and will tag the data for update, but effectively they'll be tagging the evaluated datablock for update. These are not iterated over by the depsgraph, so tagging won't have any impact. For performance & clarity we can explicitly let the tagging functions ignore evaluated datablocks. Case 4. requires some extra care. We could resolve this by only allowing the creation of new ID datablocks (most importantly Objects) from the frame-change-pre handler. This handler is called after the scene's current frame has been updated, but before the dependency graph evaluation of the new frame.

This issue was referenced by 614d70a87d

This issue was referenced by 614d70a87dc3d439d620f1df8930c37d47df3803

Added subscriber: @theres1

If I understand what you are trying to prove in your code, then changing this:

    mesh_cache_object = bpy.data.objects.get("mesh_cache")

to

     mesh_cache_object = [ob.object for ob in depsgraph.object_instances if ob.object.name == "mesh_cache"][0]

will render a static plane of vertices.

I have somewhat similar problem here #77008, though as you can see this line of code does not solve all problems.

If I understand what you are trying to prove in your code, then changing this: ``` mesh_cache_object = bpy.data.objects.get("mesh_cache") ``` to ``` mesh_cache_object = [ob.object for ob in depsgraph.object_instances if ob.object.name == "mesh_cache"][0] ``` will render a static plane of vertices. I have somewhat similar problem here #77008, though as you can see this line of code does not solve all problems.

Added subscriber: @LeoMoon

Changed status from 'Confirmed' to: 'Resolved'

Changed status from 'Resolved' to: 'Confirmed'

Ooops, got the wrong task number referenced in the commit.

Added subscriber: @OmarEmaraDev

Added subscriber: @brecht

In #71908#934122, @dr.sybren wrote:
After discussion with @Sergey we arrived at the following.

@brecht Do you have any insight in this? Any preference as to one of the alternatives? See #71908#934122 for context.

> In #71908#934122, @dr.sybren wrote: > After discussion with @Sergey we arrived at the following. @brecht Do you have any insight in this? Any preference as to one of the alternatives? See #71908#934122 for context.

Regarding #71908#934122:

More explicitly skipping depsgraph tagging of evaluated datablock sounds good to me.

Disallowing adding relations or creating new datablocks in the post handler I'm not sure about. Such changes require a depsgraph rebuild which may be relatively slow, but it's not clear to me why this is more a problem than in the pre handler. Or which specific current bug this helps with.

It's also unclear to me how these things relate to originally reported problem as described in #71908#916261.

Regarding #71908#934122: More explicitly skipping depsgraph tagging of evaluated datablock sounds good to me. Disallowing adding relations or creating new datablocks in the post handler I'm not sure about. Such changes require a depsgraph rebuild which may be relatively slow, but it's not clear to me why this is more a problem than in the pre handler. Or which specific current bug this helps with. It's also unclear to me how these things relate to originally reported problem as described in #71908#916261.

In #71908#991296, @brecht wrote:
Disallowing adding relations or creating new datablocks in the post handler I'm not sure about. Such changes require a depsgraph rebuild which may be relatively slow, but it's not clear to me why this is more a problem than in the pre handler.

If I understand BKE_scene_graph_update_for_newframe() correctly, this would be the order of things when relations change in the post-handler:

Update relations
Evaluate depsgraph
Call post handler
Update relations
Update relations (this one probably is a no-op)
Evaluate depsgraph

When done in the pre handler:

Call pre handler
Update relations
Evaluate depsgraph
Update relations

So indeed, it doesn't look like there is much of a difference, apart from the number of operations performed (and thus performance).

It's also unclear to me how these things relate to originally reported problem as described in #71908#916261.

It boils down to the question "when do we allow writing to evaluated datablocks?" So far the answer has been "never", which means that frame-change-post handlers were modifying original data bocks. This in turn led to a conflict between modifications made by the animation system and by the handler (as @Sergey explained in #71908#916261) , which caused this bug report.

If writing to evaluated datablocks is allowed, that makes it possible to first have the depsgraph evaluation produce an evaluated copy, and then have some Python code that modifies the same evaluated copy. This makes it possible to layer changes on top of each other. Obviously this only applies to the frame-change-post handler, because in the frame-change-pre handler there is no evaluated datablock to modify.

> In #71908#991296, @brecht wrote: > Disallowing adding relations or creating new datablocks in the post handler I'm not sure about. Such changes require a depsgraph rebuild which may be relatively slow, but it's not clear to me why this is more a problem than in the pre handler. If I understand `BKE_scene_graph_update_for_newframe()` correctly, this would be the order of things when relations change in the post-handler: 1. Update relations 2. Evaluate depsgraph 3. Call post handler 4. Update relations 5. Update relations (this one probably is a no-op) 6. Evaluate depsgraph When done in the pre handler: 1. Call pre handler 2. Update relations 3. Evaluate depsgraph 4. Update relations So indeed, it doesn't look like there is much of a difference, apart from the number of operations performed (and thus performance). > It's also unclear to me how these things relate to originally reported problem as described in #71908#916261. It boils down to the question "when do we allow writing to evaluated datablocks?" So far the answer has been "never", which means that frame-change-post handlers were modifying original data bocks. This in turn led to a conflict between modifications made by the animation system and by the handler (as @Sergey explained in #71908#916261) , which caused this bug report. If writing to evaluated datablocks is allowed, that makes it possible to first have the depsgraph evaluation produce an evaluated copy, and then have some Python code that modifies the same evaluated copy. This makes it possible to layer changes on top of each other. Obviously this only applies to the frame-change-post handler, because in the frame-change-pre handler there is no evaluated datablock to modify.

Added subscriber: @spiegelball

A workaround for some cases is to reassign the animated property from the evaluated datablock to the original one at the end of the script, e.g.:

...

obj.scale = obj.evaluated_get(deps).scale

EDIT:
In this case the following line enables animation of the ocean modifier:

scene.objects["mesh_cache"].modifiers["Ocean"].time = scene.objects["mesh_cache"].evaluated_get(depsgraph).modifiers["Ocean"].time

A workaround for some cases is to reassign the animated property from the evaluated datablock to the original one at the end of the script, e.g.: ``` ... obj.scale = obj.evaluated_get(deps).scale ``` **EDIT:** In this case the following line enables animation of the ocean modifier: ``` scene.objects["mesh_cache"].modifiers["Ocean"].time = scene.objects["mesh_cache"].evaluated_get(depsgraph).modifiers["Ocean"].time ```

Added subscriber: @astrid13

Could anyone tell me if this issue been fixed in any update? I am experiencing the same problem with 2.83, 2.91, 2.92 alpha. Or if anyone could kindly point me to a workaround I'd really appreciate it. Thanks

The status of this task is "Confirmed, Normal", which means that this is still an open issue.

In #71908#1004171, @spiegelball wrote:
A workaround for some cases is to reassign the animated property from the evaluated datablock to the original one at the end of the script, e.g.:
...

obj.scale = obj.evaluated_get(deps).scale
EDIT:
In this case the following line enables animation of the ocean modifier:
scene.objects["mesh_cache"].modifiers["Ocean"].time = scene.objects["mesh_cache"].evaluated_get(depsgraph).modifiers["Ocean"].time

> In #71908#1004171, @spiegelball wrote: > A workaround for some cases is to reassign the animated property from the evaluated datablock to the original one at the end of the script, e.g.: > > > ``` > ... > > obj.scale = obj.evaluated_get(deps).scale > ``` > > **EDIT:** > In this case the following line enables animation of the ocean modifier: > > ``` > scene.objects["mesh_cache"].modifiers["Ocean"].time = scene.objects["mesh_cache"].evaluated_get(depsgraph).modifiers["Ocean"].time > ```

@spiegelball Please don't repeat your own comments. This is not a support forum, but a task/issue tracker for Blender developers.

Added subscriber: @danieljabailey

I don't have the time to look at this on short notice, so unassigning myself to allow someone else to step in.

Added subscriber: @HoffmanWood

Added subscriber: @JackTheFoxOtter

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Keyframed parameters are not preserved when frame_change_post handler is used #71908