Memory not freed after rendering with 2.79a #54287

New Issue

Norbert Krieg · 2018-03-10T18:04:34+01:00

Norbert Krieg commented

2018-03-10 18:04:34 +01:00

System Information
on Debian stable arm64 with Intel chipset
with GTX 1060
rendered with or without GPU doesn't matter

Blender Version
Broken: 2.79a, latest build
Worked: 2.79

Short description of error
After rendering a scene with a lot of memory consumption while rendering the memory is not set free after rendering.
When I then load a new scene with "File-New" the memory usage seems to stay at the high value.
(In addition it seems that with the same scene the memory usage of 2.79a is higher than with 2.79.)
There's something wrong with this behaviour.

Exact steps for others to reproduce the error
Load the "fishy cat" demo scene and render it 3-5 times.
The memory usage on my system raises after every render job.
When starting with a fresh scene (without quitting Blender), the memory usage keeps the same.
After quitting Blender, the memory is set free (so no big memory leak).

Conclusion
I have to use 2.79 now, because with 2.79a I have to restart Blender every second render job - show stopper ;-(

**System Information** on Debian stable arm64 with Intel chipset with GTX 1060 rendered with or without GPU doesn't matter **Blender Version** Broken: 2.79a, latest build Worked: 2.79 **Short description of error** After rendering a scene with a lot of memory consumption while rendering the memory is not set free after rendering. When I then load a new scene with "File-New" the memory usage seems to stay at the high value. (In addition it seems that with the same scene the memory usage of 2.79a is higher than with 2.79.) There's something wrong with this behaviour. **Exact steps for others to reproduce the error** Load the "fishy cat" demo scene and render it 3-5 times. The memory usage on my system raises after every render job. When starting with a fresh scene (without quitting Blender), the memory usage keeps the same. After quitting Blender, the memory is set free (so no big memory leak). **Conclusion** I have to use 2.79 now, because with 2.79a I have to restart Blender every second render job - show stopper ;-(

Norbert Krieg commented

2018-03-10 18:04:34 +01:00

Added subscriber: @nobi08

Brecht Van Lommel commented

2018-03-10 18:32:25 +01:00

Added subscriber: @brecht

Brecht Van Lommel commented

2018-03-10 18:32:25 +01:00

I can't reproduce the problem with the fishy cat scene on Ubuntu with 2.79a.

If you have any addons enabled, try disabling them since we have had an addon cause this kind of problem before.
Where are you checking the memory usage, in the Blender top header itself? In the system monitor?
What kind of memory usage numbers are you seeing? A little bit is expected due to fragmentation, undo steps, etc, but it should be relatively small and stabilize after a few renders.
Are there any errors or warnings in the console when rendering?
How are you rendering, Shift+Z in the viewport, F12 final render, animation, .. ?

I can't reproduce the problem with the fishy cat scene on Ubuntu with 2.79a. * If you have any addons enabled, try disabling them since we have had an addon cause this kind of problem before. * Where are you checking the memory usage, in the Blender top header itself? In the system monitor? * What kind of memory usage numbers are you seeing? A little bit is expected due to fragmentation, undo steps, etc, but it should be relatively small and stabilize after a few renders. * Are there any errors or warnings in the console when rendering? * How are you rendering, Shift+Z in the viewport, F12 final render, animation, .. ?

Norbert Krieg commented

2018-03-10 19:01:18 +01:00

Changed status from 'Open' to: 'Resolved'

Norbert Krieg closed this issue

2018-03-10 19:01:18 +01:00

Norbert Krieg self-assigned this 2018-03-10 19:01:18 +01:00

Norbert Krieg commented

2018-03-10 19:01:18 +01:00

Thanks for the answer. But the hint looking at the Blender memory and the system memory usage gave the solution.
The guilty was not Blender but a system update without starting the system (which normally is not a problem if there is no kernel update).
I watched the Blender memory usage - it was ok. The system usage wasn't.
Restarted, tested... everything works fine - thanks for the fast reply.
The show can go on now :-)

Thanks for the answer. But the hint looking at the Blender memory and the system memory usage gave the solution. The guilty was not Blender but a system update without starting the system (which normally is not a problem if there is no kernel update). I watched the Blender memory usage - it was ok. The system usage wasn't. Restarted, tested... everything works fine - thanks for the fast reply. The show can go on now :-)

Norbert Krieg commented

2018-03-12 09:32:02 +01:00

Changed status from 'Resolved' to: 'Open'

Norbert Krieg reopened this issue

2018-03-12 09:32:02 +01:00

Norbert Krieg commented

2018-03-12 09:32:03 +01:00

Sorry to bother you again, but the problems seems NOT to be solved.
Again: Blender shows normal memory usage in the header information, but my system doesn't.
This is strange and appears to happen
.. with or without addons enabled,
.. with or without GPU rendering,
.. independent from starting rendering with Shift-Z, F12

with one of my bigger scenes
Memory usages:
2.79:
scene only: 2GB
while rendering: ~9GB
after rendering: back to ~3GB
reloading new scene: ~2GB
2.79a:
scene only: 2GB
while rendering: ~13GB
after rendering: back to ~11GB
reloading new scene: ~11GB

After rendering the scene memory usage in system memory stays on a high level and is not freed. Repeated renders eat up the memory of my system (with a big scene after the second rendering).
Funny thing is the 4GB higher memory consumption of the 2.79a version while rendering the same scene.

In addition I can't see any warnings/errors in the console.

Seems like being a strange problem though.
Any help appreciated.

Sorry to bother you again, but the problems seems NOT to be solved. Again: Blender shows normal memory usage in the header information, but my system doesn't. This is strange and appears to happen .. with or without addons enabled, .. with or without GPU rendering, .. independent from starting rendering with Shift-Z, F12 with one of my bigger scenes Memory usages: 2.79: scene only: 2GB while rendering: ~9GB after rendering: back to ~3GB reloading new scene: ~2GB 2.79a: scene only: 2GB while rendering: ~13GB after rendering: back to ~11GB reloading new scene: ~11GB After rendering the scene memory usage in system memory stays on a high level and is not freed. Repeated renders eat up the memory of my system (with a big scene after the second rendering). Funny thing is the 4GB higher memory consumption of the 2.79a version while rendering the same scene. In addition I can't see any warnings/errors in the console. Seems like being a strange problem though. Any help appreciated.

Norbert Krieg changed title from ~~Memory not freed after rendering on 2.79a~~ to Memory not freed after rendering with 2.79a

2018-03-12 19:18:38 +01:00

Ray molenkamp commented

2018-03-12 20:23:57 +01:00

Added subscriber: @LazyDodo

Ray molenkamp commented

2018-03-12 20:23:57 +01:00

I can't seem to repro this behavior on windows either. What exact build of 2.79a are you running? (the splash screen should have a hash on it, that would help narrowing this down)

edit: also is this a blender.org build or something from the ubuntu repositories?

I can't seem to repro this behavior on windows either. What exact build of 2.79a are you running? (the splash screen should have a hash on it, that would help narrowing this down) edit: also is this a blender.org build or something from the ubuntu repositories?

Norbert Krieg commented

2018-03-12 20:34:05 +01:00

Hash is: 8928d99270 (as of 2018-02-21 10:41)

What makes me wonder is that 2.79 works fine, but 2.79a shows this behaviour with exactly the same settings and scene.

Hash is: 8928d99270f (as of 2018-02-21 10:41) What makes me wonder is that 2.79 works fine, but 2.79a shows this behaviour with exactly the same settings and scene.

Ray molenkamp commented

2018-03-12 20:45:37 +01:00

i tried with both 2.79a and a recent master build, can't seem to trigger the issue with fishy_cat_cpu.blend maybe it's scene related? also i probably was too late editing my post, but where did you obtain the blender build? from blender.org or did it come from ubuntu's repositories?

Norbert Krieg commented

2018-03-12 21:41:09 +01:00

LazyDodo, thanks for trying.
IMO it's not scene related, because it does work normally with 2.79.
The build is from blender.org and I am not using Ubuntu but Debian stable. Since the 2.79a release I never had any problems like this one.

LazyDodo, thanks for trying. IMO it's not scene related, because it does work normally with 2.79. The build is from blender.org and I am not using Ubuntu but Debian stable. Since the 2.79a release I never had any problems like this one.

Brecht Van Lommel commented

2018-03-12 22:47:24 +01:00

It can be scene related also if it works normally in 2.79, since there can be a new bug in 2.79a that is only triggered by specific scene setups. So ideally we want statistics from a scene that we can test ourselves, to figure out where exactly the difference is.

I can see differences between Blender versions, but some randomness and delay in releasing the memory to the operating system is expected. The memory statistics in the system monitor do not tell you the actual memory that is used and available, only an approximation that can be off quite a bit. What is a problem is if re-rendering the same scene increases memory usage each time and eventually fails due to running out of memory, and I have not been able to reproduce that.

What you can try is running Blender like this, and tell us how it affects memory usage. It will make Blender release memory to the operating system quicker and eliminates some of the randomness.

MALLOC_CONF="dirty_decay_ms:0" ./blender

It can be scene related also if it works normally in 2.79, since there can be a new bug in 2.79a that is only triggered by specific scene setups. So ideally we want statistics from a scene that we can test ourselves, to figure out where exactly the difference is. I can see differences between Blender versions, but some randomness and delay in releasing the memory to the operating system is expected. The memory statistics in the system monitor do not tell you the actual memory that is used and available, only an approximation that can be off quite a bit. What is a problem is if re-rendering the same scene increases memory usage each time and eventually fails due to running out of memory, and I have not been able to reproduce that. What you can try is running Blender like this, and tell us how it affects memory usage. It will make Blender release memory to the operating system quicker and eliminates some of the randomness. ``` MALLOC_CONF="dirty_decay_ms:0" ./blender ```

YAFU commented

2018-03-13 01:09:51 +01:00

Added subscriber: @YAFU

YAFU commented

2018-03-13 01:09:51 +01:00

I am able to reproduce the problem with Blender 2.79a official release and this scene:
High RAM scene.blend

My system: Kubuntu Linux 18.04 64bits - GTX 960 4GB - i7 3770 - 15.6GB RAM available.

Steps (All the procedure is without closing Blender):

*2.79a
Open High RAM scene.blend scene, Render image. After finished System Monitor reports Blender still uses about 12.6GB.

File > New > Reload Start-UP file. Turn to Cycles. Render image (default cube). After finished System Monitor reports Blender still uses about 11.9GB.

File > Open Recent > High RAM scene.blend. render Image. After finished System Monitor reports Blender still uses about 12.8GB

So apparently memory is never freed, but apparently it is not cumulative either.

*2.79
The behavior is different, while render scene simply it reaches a peak of 13GB, but frees the memory when it ends.

I am able to reproduce the problem with Blender 2.79a official release and this scene: [High RAM scene.blend](https://archive.blender.org/developer/F2439528/High_RAM_scene.blend) My system: Kubuntu Linux 18.04 64bits - GTX 960 4GB - i7 3770 - 15.6GB RAM available. Steps (All the procedure is without closing Blender): *2.79a Open High RAM scene.blend scene, Render image. After finished System Monitor reports Blender still uses about 12.6GB. File > New > Reload Start-UP file. Turn to Cycles. Render image (default cube). After finished System Monitor reports Blender still uses about 11.9GB. File > Open Recent > High RAM scene.blend. render Image. After finished System Monitor reports Blender still uses about 12.8GB So apparently memory is never freed, but apparently it is not cumulative either. *2.79 The behavior is different, while render scene simply it reaches a peak of 13GB, but frees the memory when it ends.

Ray molenkamp commented

2018-03-13 01:32:47 +01:00

i had to dial down the particle system a tiny bit because of ram issues but this is the memory curve i got on windows with 2.79a with that file

i had to dial down the particle system a tiny bit because of ram issues but this is the memory curve i got on windows with 2.79a with that file ![image.png](https://archive.blender.org/developer/F2439555/image.png)

Brecht Van Lommel commented

2018-03-13 01:41:27 +01:00

Added subscriber: @Sergey

Brecht Van Lommel commented

2018-03-13 01:41:27 +01:00

@YAFU, does running Blender 2.79a like this cause the reported memory usage to be reduced after rendering?

MALLOC_CONF="dirty_decay_ms:0" ./blender

@Sergey, did the jemalloc version change for 2.79a?

@YAFU, does running Blender 2.79a like this cause the reported memory usage to be reduced after rendering? ``` MALLOC_CONF="dirty_decay_ms:0" ./blender ``` @Sergey, did the jemalloc version change for 2.79a?

YAFU commented

2018-03-13 01:47:54 +01:00

@LazyDodo. Maybe just Linux issue? My system monitor graphic with Blender 2.79a is basically a permanent horizontal line at approximately 13.5 GB :)
Just in case, using 4.15 linux kernel here.

@brecht Van Lommel. I had forgotten to try it. Running Blender 2.79a with "MALLOC_CONF="dirty_decay_ms:0" ./blender" it works fine, memory is freed.

@LazyDodo. Maybe just Linux issue? My system monitor graphic with Blender 2.79a is basically a permanent horizontal line at approximately 13.5 GB :) Just in case, using 4.15 linux kernel here. @brecht Van Lommel. I had forgotten to try it. Running Blender 2.79a with "MALLOC_CONF="dirty_decay_ms:0" ./blender" it works fine, memory is freed.

Norbert Krieg commented

2018-03-13 08:01:54 +01:00

@brecht also for me: "MALLOC_CONF="dirty_decay_ms:0" blender" works fine.
No cumulation of allocated memory on system monitor/blender monitor. Memory is freed after rendering, loading new scene.
Seems to work fine with that env variable setting.

@brecht also for me: "MALLOC_CONF="dirty_decay_ms:0" blender" works fine. No cumulation of allocated memory on system monitor/blender monitor. Memory is freed after rendering, loading new scene. Seems to work fine with that env variable setting.

Sergey Sharybin commented

2018-03-13 09:28:50 +01:00

@brecht, jemalloc indeed changed for 2.79a. The reason for that was that previous update introduced rather dramatic slowdown for things like Draw Manager behavior in 2.8. This slowdown was fixed by jemalloc-5.0.1, but seems it introduced some other issues?

Brecht Van Lommel commented

2018-03-13 23:34:57 +01:00

It seems the dirty page purging totally changed:
https://github.com/jemalloc/jemalloc/releases/tag/5.0.0
https://github.com/jemalloc/jemalloc/issues/325

Purging dirty pages is slower than before when opening a big .blend and then doing file > new, but it still happens after some time in that case. This can be explained by the default 10s delay:
http://jemalloc.net/jemalloc.3.html#opt.dirty_decay_ms

However for rendering we allocate memory in different threads, and according to this comment that can be problematic:

(a lot of your arenas only have one thread in them -- if that thread is sleeping or otherwise not interacting with the allocator much, it might never get far enough into the arena code to purge).

And indeed, if we render a simple scene afterwards a few times it wakes up the render threads, and memory goes down a bit each time. Similar behavior happens for Cycles, Blender Internal, baking, etc. The suggested solution seems to be enabling jemalloc background threads:

diff --git a/source/creator/creator.c b/source/creator/creator.c
index a59a45f..15b4169 100644
--- a/source/creator/creator.c
+++ b/source/creator/creator.c
@@ -201,6 +201,10 @@ char **environ = NULL;
 - endif
 - endif
 
+/* If jemalloc is used, it reads this global variable and enables background
+ * threads to purge dirty pages. Otherwise we release memory too slowly. */
+const char *malloc_conf = "background_thread:true";
+
 /**
  * Blender's main function responsibilities are:
  * - setup subsystems.

With that option it seems to purge all the dirty pages within about 10s after rendering as expected. It's not entirely clear to me if this would solve all potential problems though. Suppose we allocate and free a bunch of memory in one thread, and then do the same in another thread soon afterwards. Can it now run out of memory if the sum of what the two threads allocated is too high? The background threads are not going to be able to purge the pages quick enough, so will jemalloc do it in some other way to avoid running out of memory?

It seems the dirty page purging totally changed: https://github.com/jemalloc/jemalloc/releases/tag/5.0.0 https://github.com/jemalloc/jemalloc/issues/325 Purging dirty pages is slower than before when opening a big .blend and then doing file > new, but it still happens after some time in that case. This can be explained by the default 10s delay: http://jemalloc.net/jemalloc.3.html#opt.dirty_decay_ms However for rendering we allocate memory in different threads, and according to [this comment ](https://github.com/jemalloc/jemalloc/issues/956#issuecomment-316171166) that can be problematic: > (a lot of your arenas only have one thread in them -- if that thread is sleeping or otherwise not interacting with the allocator much, it might never get far enough into the arena code to purge). And indeed, if we render a simple scene afterwards a few times it wakes up the render threads, and memory goes down a bit each time. Similar behavior happens for Cycles, Blender Internal, baking, etc. The suggested solution seems to be enabling jemalloc background threads: ``` diff --git a/source/creator/creator.c b/source/creator/creator.c index a59a45f..15b4169 100644 --- a/source/creator/creator.c +++ b/source/creator/creator.c @@ -201,6 +201,10 @@ char **environ = NULL; - endif - endif +/* If jemalloc is used, it reads this global variable and enables background + * threads to purge dirty pages. Otherwise we release memory too slowly. */ +const char *malloc_conf = "background_thread:true"; + /** * Blender's main function responsibilities are: * - setup subsystems. ``` With that option it seems to purge all the dirty pages within about 10s after rendering as expected. It's not entirely clear to me if this would solve all potential problems though. Suppose we allocate and free a bunch of memory in one thread, and then do the same in another thread soon afterwards. Can it now run out of memory if the sum of what the two threads allocated is too high? The background threads are not going to be able to purge the pages quick enough, so will jemalloc do it in some other way to avoid running out of memory?

Sergey Sharybin commented

2018-03-15 11:01:02 +01:00

@brecht, i propose the following:

We make 2.79b with previous version of jemalloc, just so we are safe and everything.
We use your solution for master branch and see how that behaves. We shouldn't be locking to an older jemalloc versions, lots of distros are moving to a newer versions now.

I'm also quite sure jemalloc is used in other software, including Firefox with all this separate threads for JIT compilation and such. Can totally check how it's used there or ask developers.

@brecht, i propose the following: - We make 2.79b with previous version of jemalloc, just so we are safe and everything. - We use your solution for master branch and see how that behaves. We shouldn't be locking to an older jemalloc versions, lots of distros are moving to a newer versions now. I'm also quite sure jemalloc is used in other software, including Firefox with all this separate threads for JIT compilation and such. Can totally check how it's used there or ask developers.

Brecht Van Lommel commented

2018-03-15 21:27:41 +01:00

Sounds like a good plan.

Firefox did not upgrade to jemalloc 4 or 5, they're using a customized earlier version of jemalloc.
https://bugzilla.mozilla.org/show_bug.cgi?id=1363992

Sounds like a good plan. Firefox did not upgrade to jemalloc 4 or 5, they're using a customized earlier version of jemalloc. https://bugzilla.mozilla.org/show_bug.cgi?id=1363992

Sergey Sharybin commented

2018-03-23 21:20:09 +01:00

@brecht, 2.79b builds are done. If you can verify they're fine it'll be great! The buildbot is back to jemalloc 5.0.1, so this issue we need to address in one way or another.

Sergey Sharybin commented

2018-03-26 12:28:18 +02:00

Just verified with Campbell. The issue is solved in 2.79b by using older jemalloc.

Buildbot is on newer jemalloc, so let's see what we can do there.

@brecht, Using MALLOC_CONF="dirty_decay_ms:0" solves the issue. Tried your patch, but this gives me:

<jemalloc>: Error in dlsym(RTLD_NEXT, "pthread_create")
Aborted

on the very startup.

Just verified with Campbell. The issue is solved in 2.79b by using older jemalloc. Buildbot is on newer jemalloc, so let's see what we can do there. @brecht, Using `MALLOC_CONF="dirty_decay_ms:0"` solves the issue. Tried your patch, but this gives me: ``` <jemalloc>: Error in dlsym(RTLD_NEXT, "pthread_create") Aborted ``` on the very startup.

Brecht Van Lommel commented

2018-03-27 18:09:18 +02:00

There's a bug report about that error here, I guess you are using 5.0.1 with the fix already?
https://github.com/jemalloc/jemalloc/issues/907

It seems there are multiple issues depending on static / shared libraries and linking order, and not all of them may be solved. It worked for me with a static jemalloc 5.0.1 and no changes to the build system, not sure how your setup is different.

I think dirty_decay_ms:0 is not intended for production use and performance is expected to be poor, not sure how much it matters in practice but probably it's not a good idea to use it.

There's a bug report about that error here, I guess you are using 5.0.1 with the fix already? https://github.com/jemalloc/jemalloc/issues/907 It seems there are multiple issues depending on static / shared libraries and linking order, and not all of them may be solved. It worked for me with a static jemalloc 5.0.1 and no changes to the build system, not sure how your setup is different. I think `dirty_decay_ms:0` is not intended for production use and performance is expected to be poor, not sure how much it matters in practice but probably it's not a good idea to use it.

Sergey Sharybin commented

2018-03-28 10:08:15 +02:00

@brecht, i'm dynamically linking against libjemalloc v5.0.1 here. Did you talk to those folks about how to make newer jemalloc usable for Blender? :)

Christian Hubert commented

2018-05-28 07:35:26 +02:00

Added subscriber: @christian

Lukas commented

2018-05-31 13:15:42 +02:00

Added subscriber: @lsstratmann

Lukas commented

2018-05-31 13:15:42 +02:00

test.blend

I can confirm this or a similar issue with 2.80 (build "2.80-f8908f0d434-linux-glibc219-x86_64"). When I try to render about 300 frames of the default cube animation above, my system memory usage increases by about 7 GiB while Blender itself only reports about 100 MB of memory being used.
The output format in this file is set to "Ffmpeg video"; my ffmpeg version is 4.0.

[test.blend](https://archive.blender.org/developer/F3530896/test.blend) I can confirm this or a similar issue with 2.80 (build "2.80-f8908f0d434-linux-glibc219-x86_64"). When I try to render about 300 frames of the default cube animation above, my system memory usage increases by about 7 GiB while Blender itself only reports about 100 MB of memory being used. The output format in this file is set to "Ffmpeg video"; my ffmpeg version is 4.0.

Norbert Krieg commented

2018-05-31 14:15:00 +02:00

I changed the render samples to 1 and render size to 50% to speed up things.
I started rendering. System memory consumption raises to >16GB after short time with blender displays memory usage around 70MB.
I can confirm Lukas' observation. I used blender-2.80-fbd614f1faf-linux-glibc219-x86_64 on a Debian stable system with a GTX1060.
Seems as if the jmalloc problem is here again??

I changed the render samples to 1 and render size to 50% to speed up things. I started rendering. System memory consumption raises to >16GB after short time with blender displays memory usage around 70MB. I can confirm Lukas' observation. I used blender-2.80-fbd614f1faf-linux-glibc219-x86_64 on a Debian stable system with a GTX1060. Seems as if the jmalloc problem is here again??

Patrice commented

2018-09-18 13:22:59 +02:00

Added subscriber: @Metal3d

Patrice commented

2018-09-18 13:22:59 +02:00

Seems to be related to what I reported here https://developer.blender.org/T56825 I'll try the MALLOC option

Patrice commented

2018-09-18 13:45:14 +02:00

I confirm that MALLOC_CONF changed the behavior on my computer with 20G RAM and 1060 ti (3g) NVidia card on Fedora 28. Memory seems to be cleaned and I do not have computer freeze after several renders.

YAFU commented

2018-09-18 14:43:06 +02:00

I still have the problem in 2.79 and 2.8 from Buildbot builds. Compiling Blender by myself the problem does not happen. I suppose that is due to the version of jemalloc installed in my system (3.6.0)

blender-admin commented

2018-10-03 11:10:08 +02:00

This issue was referenced by d2da3af073

This issue was referenced by d2da3af073a63b6ae25119ebbd2e0d3c9a1b6823

Brecht Van Lommel commented

2018-10-03 12:12:05 +02:00

Changed status from 'Open' to: 'Resolved'

Brecht Van Lommel closed this issue

2018-10-03 12:12:05 +02:00

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Memory not freed after rendering with 2.79a #54287