Add Support of PointCaches to Library Overrides #82503

New Issue

Daniel Salazar · 2020-11-08T06:51:28+01:00

Daniel Salazar commented

2020-11-08 06:51:28 +01:00

Support baking simulations into PointCaches for library overrides.

Milestone 1 - Minimal Viable Product - 2.92

Support baking existing cache slots on disk (59910f7217, 50ccf346f0, 13bcb000fa, b073f58a8e).

Milestone 2 - Support memory cache - ???

Support baking existing cache slots in memory (this requires some specific handling during application of overrides step).

Milestone 3 - Potential Extensions - ???
Those will require #82160 (LibOverrides - Refactor how diffing of RNA collections is handled, and extend diffing possibilities.) to be tackled first.

Support adding new cache slots to a simulation.

Original Report

System Information
Operating system: Windows-10-10.0.19041-SP0 64 Bits
Graphics card: GeForce GTX 1080 with Max-Q Design/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 457.09

Blender Version
Broken: version: 2.92.0 Alpha, branch: master, commit date: 2020-11-06 19:20, hash: d2c102060d

Short description of error
Cloth sim bakes work great on library overrides until you save and reload. Then the bake is lost.

Exact steps for others to reproduce the error
Cloth.blend
Library Override.blend
Download both files
Open "Library Override.blend"
Press Bake
Save
Reload

Support baking simulations into PointCaches for library overrides. **Milestone 1 - Minimal Viable Product - 2.92** - [x] Support baking **existing** cache slots on disk (59910f7217, 50ccf346f0, 13bcb000fa, b073f58a8e). **Milestone 2 - Support memory cache - ???** - [ ] Support baking **existing** cache slots in memory *(this requires some specific handling during application of overrides step)*. **Milestone 3 - Potential Extensions - ???** *Those will require #82160 (LibOverrides - Refactor how diffing of RNA collections is handled, and extend diffing possibilities.) to be tackled first.* - [ ] Support **adding** new cache slots to a simulation. -------------- *Original Report* **System Information** Operating system: Windows-10-10.0.19041-SP0 64 Bits Graphics card: GeForce GTX 1080 with Max-Q Design/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 457.09 **Blender Version** Broken: version: 2.92.0 Alpha, branch: master, commit date: 2020-11-06 19:20, hash: `d2c102060d` **Short description of error** Cloth sim bakes work great on library overrides until you save and reload. Then the bake is lost. **Exact steps for others to reproduce the error** [Cloth.blend](https://archive.blender.org/developer/F9233717/Cloth.blend) [Library Override.blend](https://archive.blender.org/developer/F9233718/Library_Override.blend) Download both files Open "Library Override.blend" Press Bake Save Reload

Daniel Salazar commented

2020-11-08 06:51:28 +01:00

Added subscriber: @zanqdo

Philipp Oeser commented

2020-11-09 13:44:17 +01:00

Added subscribers: @mont29, @lichtwerk

Philipp Oeser commented

2020-11-09 13:44:17 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Philipp Oeser commented

2020-11-09 13:44:17 +01:00

Can confirm.

Not totally sure about how much is supported here, think @mont29 would know best.

Can confirm. Not totally sure about how much is supported here, think @mont29 would know best.

Bastien Montagne commented

2020-11-09 17:04:54 +01:00

Thanks for the report, but this is indeed not a bug, and is not intended to work currently. We need to find a proper way to swap back local point caches and linked ones, but this is not straightforward as there may be differences (especially due to modifiers).

So I believe the best way to deal with this would be to use Disk cache for the time being.

Thanks for the report, but this is indeed not a bug, and is not intended to work currently. We need to find a proper way to swap back local point caches and linked ones, but this is not straightforward as there may be differences (especially due to modifiers). So I believe the best way to deal with this would be to use Disk cache for the time being.

Bastien Montagne commented

2020-11-09 17:04:58 +01:00

Removed subscriber: @mont29

Bastien Montagne commented

2020-11-09 17:05:05 +01:00

Added subscriber: @mont29

Daniel Salazar commented

2020-11-09 23:01:57 +01:00

@mont29 Thanks but how does it work?

I tried enabling the disk cache option in the character file. But in the linked file the setting will be disabled and there's nothing I can do to enable it.

@mont29 Thanks but how does it work? I tried enabling the disk cache option in the character file. But in the linked file the setting will be disabled and there's nothing I can do to enable it. ![image.png](https://archive.blender.org/developer/F9248127/image.png)

Bastien Montagne commented

2020-11-12 19:20:20 +01:00

For the records, I've been trying to understand what is going on with those pointcaches, so far no luck...

Have a basic patch to enable some of their properties for override, and get better control over the operators (Bake & co) in override cases, but there is a very weird thing happening here.

Those properties do not get overridden properly, their values keep changing randomly... There must be some dark RNA magic happening here. :(

For the records, I've been trying to understand what is going on with those pointcaches, so far no luck... Have a basic patch to enable some of their properties for override, and get better control over the operators (Bake & co) in override cases, but there is a very weird thing happening here. Those properties do not get overridden properly, their values keep changing randomly... There must be some dark RNA magic happening here. :(

Daniel Salazar commented

2020-11-13 02:16:40 +01:00

Thanks for taking a look!

blender-admin commented

2020-11-13 14:12:03 +01:00

This issue was referenced by 59910f7217

This issue was referenced by 59910f721721117108c45a6b961812beb3955625

blender-admin commented

2020-11-13 14:20:10 +01:00

This issue was referenced by 50ccf346f0

This issue was referenced by 50ccf346f0b8bbf91811a88ba5f31a85dcab8467

Bastien Montagne commented

2020-11-13 14:43:53 +01:00

Bake to disk should now work in master (has been tough to track down, we had a pile of non-obvious issues to solve there...).

Supporting bake in memory is for later, this is trickier to do properly.

Bake to disk should now work in master (has been tough to track down, we had a pile of non-obvious issues to solve there...). Supporting bake in memory is for later, this is trickier to do properly.

Daniel Salazar commented

2020-11-16 08:38:48 +01:00

I gave it a test and it did work on 2 out of 3 characters strangely enough! I have simplified as much as I could and here are the test files:

Original.blend
Library Override.blend

Download both files to the same directory
Open Library override
Bake
Save
Reload

I gave it a test and it did work on 2 out of 3 characters strangely enough! I have simplified as much as I could and here are the test files: [Original.blend](https://archive.blender.org/developer/F9299768/Original.blend) [Library Override.blend](https://archive.blender.org/developer/F9299767/Library_Override.blend) Download both files to the same directory Open Library override Bake Save Reload

Daniel Salazar commented

2020-11-16 09:59:16 +01:00

Actually I just lost another character's bake I don't know how. So I guess it's affecting all objects.

blender-admin commented

2020-11-16 12:29:21 +01:00

This issue was referenced by 13bcb000fa

This issue was referenced by 13bcb000fa81effa3c8302b2faba853c93a6c573

Bastien Montagne commented

2020-11-16 14:49:13 +01:00

In #82503#1055339, @zanqdo wrote:
I gave it a test and it did work on 2 out of 3 characters strangely enough! I have simplified as much as I could and here are the test files:

Original.blend
Library Override.blend

Download both files to the same directory
Open Library override
Bake
Save
Reload

Should be fixed "good enough" by 13bcb000fa

> In #82503#1055339, @zanqdo wrote: > I gave it a test and it did work on 2 out of 3 characters strangely enough! I have simplified as much as I could and here are the test files: > > [Original.blend](https://archive.blender.org/developer/F9299768/Original.blend) > [Library Override.blend](https://archive.blender.org/developer/F9299767/Library_Override.blend) > > Download both files to the same directory > Open Library override > Bake > Save > Reload Should be fixed "good enough" by 13bcb000fa

Bastien Montagne changed title from ~~Locally baked cloth simulations on library overrides are lost after reload~~ to Add Support of PointCaches to Library Overrides

2020-11-17 20:58:27 +01:00

Bastien Montagne self-assigned this 2020-11-17 20:58:27 +01:00

Jon Friesen commented

2021-01-05 14:27:07 +01:00

Added subscriber: @Threedio

Jon Friesen commented

2021-01-05 14:27:07 +01:00

Hi Bastien,

If I'm reading 13bcb000fa: "Fix broken pointcache on disk in some cases" correctly, the fix is committed and should be in the latest 2.92 alpha builds, am I correct? But I re-tested this bug in the Jan 5 build of 2.92 alpha (blender-2.92.0-9b17e71c23a9) and the Jan 5 build of the experimental-build branch (experimental-build-blender-2.92.0-123fd3d5c446) and am having the same issue originally reported. I'll attach my {F9551380}test files (without the bake cache to save space); here's how I reproduced the error:

create model/hairy.cube.blend

delete camera and light, keep default collection & cube
add particle settings and switch to hair
enable hair dynamics
change cache to Disk Cache and de-select Use Library Path
save and close

create scene/bugtest.scene.blend

delete camera, light and cube
File -> Link; browse to model file --> Collection --> Collection, and link the Collection
select the collection and Make Library Override
select the cube and navigate to particles --> cache
bake. Confirm that it says "250 frames on disk"

observe the bug

close the file
open scene/bugtest.scene.blend again
select the cube and navigate to particles --> cache
Confirm that it says "0 frames on disk"

other observations

if I Make Local the object, the issue cannot be reproduced (this was already known but I just wanted to re-confirm that the error is specific to library overrides)
the issue can be reproduced if I link in the individual cube object (instead of the collection)

I am new to the blender dev forums so I hope I've approached the test correctly.

Hi Bastien, If I'm reading 13bcb000fa: "Fix broken pointcache on disk in some cases" correctly, the fix is committed and should be in the latest 2.92 alpha builds, am I correct? But I re-tested this bug in the Jan 5 build of 2.92 alpha (blender-2.92.0-9b17e71c23a9) and the Jan 5 build of the experimental-build branch (experimental-build-blender-2.92.0-123fd3d5c446) and am having the same issue originally reported. I'll attach my {[F9551380](https://archive.blender.org/developer/F9551380/bugtest.13bcb000fa81.zip)}test files (without the bake cache to save space); here's how I reproduced the error: create model/hairy.cube.blend - delete camera and light, keep default collection & cube - add particle settings and switch to hair - enable hair dynamics - change cache to Disk Cache and de-select Use Library Path - save and close create scene/bugtest.scene.blend - delete camera, light and cube - File -> Link; browse to model file --> Collection --> Collection, and link the Collection - select the collection and Make Library Override - select the cube and navigate to particles --> cache - bake. Confirm that it says "250 frames on disk" observe the bug - close the file - open scene/bugtest.scene.blend again - select the cube and navigate to particles --> cache - Confirm that it says "0 frames on disk" other observations - if I Make Local the object, the issue cannot be reproduced (this was already known but I just wanted to re-confirm that the error is specific to library overrides) - the issue can be reproduced if I link in the individual cube object (instead of the collection) I am new to the blender dev forums so I hope I've approached the test correctly.

Bastien Montagne commented

2021-01-05 15:24:41 +01:00

@Threedio besides the fact that you forget the Save step at the end of your create scene/bugtest.scene.blend section, did those steps and it works for me?

@Threedio besides the fact that you forget the `Save` step at the end of your `create scene/bugtest.scene.blend` section, did those steps and it works for me?

Jon Friesen commented

2021-01-05 18:58:48 +01:00

Hi Bastien,

Ha yes I definitely saved my file after baking the frames. That is peculiar that it is working for you and not for me -- which release are you testing with?

I'm going to try it on my linux box and see if I get the same results, as an additional test.

Hi Bastien, Ha yes I definitely saved my file after baking the frames. That is peculiar that it is working for you and not for me -- which release are you testing with? I'm going to try it on my linux box and see if I get the same results, as an additional test.

Jon Friesen commented

2021-01-05 19:35:32 +01:00

I was able to reproduce the issue on my linux box too. Attached is a video of me reproducing the error on my Windows box -- hopefully I am not embarrassing myself with some kind of user error. Untitled1.mp4

I was able to reproduce the issue on my linux box too. Attached is a video of me reproducing the error on my Windows box -- hopefully I am not embarrassing myself with some kind of user error. [Untitled1.mp4](https://archive.blender.org/developer/F9551755/Untitled1.mp4)

Jon Friesen commented

2021-01-09 14:57:58 +01:00

To further illustrate the issue, I added a second linked library override cube to my scene, and for the second cube, I did Make Local --> Object and Data. Side by side I can show that the local object retains its cache but the linked object does not. I'll attach the blend files here (bugtest.13bcb000fa81-2021-01-08.zip) and a screencast depicting the issue.

bugtest2.mp4

To further illustrate the issue, I added a second linked library override cube to my scene, and for the second cube, I did Make Local --> Object and Data. Side by side I can show that the local object retains its cache but the linked object does not. I'll attach the blend files here ([bugtest.13bcb000fa81-2021-01-08.zip](https://archive.blender.org/developer/F9557113/bugtest.13bcb000fa81-2021-01-08.zip)) and a screencast depicting the issue. [bugtest2.mp4](https://archive.blender.org/developer/F9557115/bugtest2.mp4)

Bastien Montagne commented

2021-01-22 11:21:31 +01:00

Arf, spent quiet some time on this, but this is in fact only a display issue (message is not updated when reading overrides, but cache is indeed there, and is used perfectly).

Found other issues regarding hair cache settings though, thanks to that same unique pointcache being exposed in four different RNA paths... sigh

Arf, spent quiet some time on this, but this is in fact only a display issue (message is not updated when reading overrides, but cache is indeed there, and is used perfectly). Found other issues regarding hair cache settings though, thanks to that same unique pointcache being exposed in **four** different RNA paths... *sigh*

Jon Friesen commented

2021-01-22 12:39:15 +01:00

Rats. In an animated feature I'm working on, I have a much more complicated scene, and when I bake and render the animation, the hair dynamics of my library override characters animate correctly, but if I bake, save and close, and render from the command line, the hair dynamics are lost. I thought I had targetted the issue, but you are right, I was on the wrong track with this display bug. I will try again to distill my issue down to a simple test case. Sorry for the red herring!

Bastien Montagne commented

2021-01-22 15:35:23 +01:00

UI message not being updated shall be fixed now (be7106a974). Also fixed the hair-specific issues in cd8893d446.

UI message not being updated shall be fixed now (be7106a974). Also fixed the hair-specific issues in cd8893d446.

炎臻 commented

2021-05-06 05:31:51 +02:00

Added subscriber: @PrettyFireNOI7

Stephen Hamacek commented

2021-09-20 08:02:12 +02:00

Added subscriber: @StephenHamacek

Stephen Hamacek commented

2021-09-20 08:02:12 +02:00

I'm trying to bake cloth/disk cache on an overridden character, but sadly just get a hard crash when I hit bake. Shouldn't be a computer performance issue as I can still bake using the old 'pin the object panel' method that I've been using for years with proxy characters.

I also tried making the garment object local after creating the character override. This gave me (visually at least) access to all the cloth mod settings, which is my holy grail for per-scene baking. But sadly also a hard crash on 'bake'.

(Mac 2.93 LTS)

Is this still being looked at and should I try getting a .blend file together? Many thanks

I'm trying to bake cloth/disk cache on an overridden character, but sadly just get a hard crash when I hit bake. Shouldn't be a computer performance issue as I can still bake using the old 'pin the object panel' method that I've been using for years with proxy characters. I also tried making the garment object local after creating the character override. This gave me (visually at least) access to all the cloth mod settings, which is my holy grail for per-scene baking. But sadly also a hard crash on 'bake'. (Mac 2.93 LTS) Is this still being looked at and should I try getting a .blend file together? Many thanks

Stephen Hamacek commented

2021-09-22 14:46:33 +02:00

I get a seg fault 11 on these hard crashes, thread 0. Here's the first part of the crash report. I'd love to know how people are managing to bake, maybe a mac thing. I'll try to put together a stripped down .blend and submit a bug report

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_CRASH (SIGSEGV)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   Blender [6728]

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   org.blenderfoundation.blender 	0x000000010fc4b9db psys_cache_paths + 2299
1   org.blenderfoundation.blender 	0x000000010fc63968 psys_update_path_cache + 440
2   org.blenderfoundation.blender 	0x000000010fc61e63 particle_system_update + 10035
3   org.blenderfoundation.blender 	0x00000001101cd5bf deformVerts + 591
4   org.blenderfoundation.blender 	0x000000010faa40b3 mesh_calc_modifiers(Depsgraph*, Scene*, Object*, int, bool, CustomData_MeshMasks const*, int, bool, bool, Mesh**, Mesh**, GeometrySet**) + 1011
5   org.blenderfoundation.blender 	0x000000010faa39c5 mesh_build_data(Depsgraph*, Scene*, Object*, CustomData_MeshMasks const*, bool) + 133
6   org.blenderfoundation.blender 	0x000000010faa2a25 makeDerivedMesh + 133
7   org.blenderfoundation.blender 	0x000000010fc37838 BKE_object_handle_data_update + 1048
8   org.blenderfoundation.blender 	0x000000010fc37ca4 BKE_object_eval_uber_data + 52
9   org.blenderfoundation.blender 	0x00000001172919c6 blender::deg::(anonymous namespace)::deg_task_run_func(TaskPool*, void*) + 134
10  org.blenderfoundation.blender 	0x000000010ff9386b tbb::interface7::internal::isolate_within_arena(tbb::interface7::internal::delegate_base&, long) + 75
11  org.blenderfoundation.blender 	0x00000001193e8152 tbb::internal::function_task<Task>::execute() + 50
12  org.blenderfoundation.blender 	0x000000010ffa0369 tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop(tbb::internal::context_guard_helper<false>&, tbb::task*, long) + 393
13  org.blenderfoundation.blender 	0x000000010ff9f9b1 tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all(tbb::task&, tbb::task*) + 225
14  org.blenderfoundation.blender 	0x0000000111f95788 tbb::internal::task_group_base::wait() + 24
15  org.blenderfoundation.blender 	0x00000001193e7d54 BLI_task_pool_work_and_wait + 516
16  org.blenderfoundation.blender 	0x00000001172917b1 blender::deg::deg_evaluate_on_refresh(blender::deg::Depsgraph*) + 657
17  org.blenderfoundation.blender 	0x000000010fc83702 BKE_scene_graph_update_for_newframe_ex + 114
18  org.blenderfoundation.blender 	0x000000010ffab179 wm_event_do_handlers + 553
19  org.blenderfoundation.blender 	0x000000010ffa42c0 WM_main + 32
20  org.blenderfoundation.blender 	0x000000010fa9225b main + 907
21  libdyld.dylib                 	0x00007fff58881015 start + 1

I get a seg fault 11 on these hard crashes, thread 0. Here's the first part of the crash report. I'd love to know how people are managing to bake, maybe a mac thing. I'll try to put together a stripped down .blend and submit a bug report ``` Crashed Thread: 0 Dispatch queue: com.apple.main-thread Exception Type: EXC_CRASH (SIGSEGV) Exception Codes: 0x0000000000000000, 0x0000000000000000 Exception Note: EXC_CORPSE_NOTIFY Termination Signal: Segmentation fault: 11 Termination Reason: Namespace SIGNAL, Code 0xb Terminating Process: Blender [6728] Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 org.blenderfoundation.blender 0x000000010fc4b9db psys_cache_paths + 2299 1 org.blenderfoundation.blender 0x000000010fc63968 psys_update_path_cache + 440 2 org.blenderfoundation.blender 0x000000010fc61e63 particle_system_update + 10035 3 org.blenderfoundation.blender 0x00000001101cd5bf deformVerts + 591 4 org.blenderfoundation.blender 0x000000010faa40b3 mesh_calc_modifiers(Depsgraph*, Scene*, Object*, int, bool, CustomData_MeshMasks const*, int, bool, bool, Mesh**, Mesh**, GeometrySet**) + 1011 5 org.blenderfoundation.blender 0x000000010faa39c5 mesh_build_data(Depsgraph*, Scene*, Object*, CustomData_MeshMasks const*, bool) + 133 6 org.blenderfoundation.blender 0x000000010faa2a25 makeDerivedMesh + 133 7 org.blenderfoundation.blender 0x000000010fc37838 BKE_object_handle_data_update + 1048 8 org.blenderfoundation.blender 0x000000010fc37ca4 BKE_object_eval_uber_data + 52 9 org.blenderfoundation.blender 0x00000001172919c6 blender::deg::(anonymous namespace)::deg_task_run_func(TaskPool*, void*) + 134 10 org.blenderfoundation.blender 0x000000010ff9386b tbb::interface7::internal::isolate_within_arena(tbb::interface7::internal::delegate_base&, long) + 75 11 org.blenderfoundation.blender 0x00000001193e8152 tbb::internal::function_task<Task>::execute() + 50 12 org.blenderfoundation.blender 0x000000010ffa0369 tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop(tbb::internal::context_guard_helper<false>&, tbb::task*, long) + 393 13 org.blenderfoundation.blender 0x000000010ff9f9b1 tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all(tbb::task&, tbb::task*) + 225 14 org.blenderfoundation.blender 0x0000000111f95788 tbb::internal::task_group_base::wait() + 24 15 org.blenderfoundation.blender 0x00000001193e7d54 BLI_task_pool_work_and_wait + 516 16 org.blenderfoundation.blender 0x00000001172917b1 blender::deg::deg_evaluate_on_refresh(blender::deg::Depsgraph*) + 657 17 org.blenderfoundation.blender 0x000000010fc83702 BKE_scene_graph_update_for_newframe_ex + 114 18 org.blenderfoundation.blender 0x000000010ffab179 wm_event_do_handlers + 553 19 org.blenderfoundation.blender 0x000000010ffa42c0 WM_main + 32 20 org.blenderfoundation.blender 0x000000010fa9225b main + 907 21 libdyld.dylib 0x00007fff58881015 start + 1 ```

Bastien Montagne commented

2021-12-28 09:25:50 +01:00

@StephenHamacek ah missed your messages, please report a new bug if this is still an issue, comments in design or TODO tasks is not the proper way to do so and might be missed. ;)

blender-admin commented

2022-04-01 12:41:59 +02:00

This issue was referenced by b073f58a8e

This issue was referenced by b073f58a8e96ab3224c67550672a051e61a718e8

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Add Support of PointCaches to Library Overrides #82503