Python/BMesh: Add/removing textures between renders causes memory leak/ram crash #55954

New Issue

Charles West · 2018-07-12T21:06:54+02:00

Charles West commented

2018-07-12 21:06:54 +02:00

System Information
Operating system and graphics card
Ubuntu 18.04, Nvidia GTX 960M

Blender Version
Broken: blender-2.80-17bc0567268-linux-glibc219-x86_64

Working Blender Version
Blender 2.79b stable (ubuntu repository) with cycles

Short description of error
If you run the attached code and target it a directory with a large number of images, it will repeatedly:

Load an image from file
Set the image to be the content of a "Image Texture" node
Delete/remove the image that use to be the image texture node
Render

As it processes images, blender takes more and more memory. Eventually it will crash. If you leave out step 4, then it will not.

Exact steps for others to reproduce the error
Based on a (as simple as possible) attached .blend file with minimum amount of steps

Import python code, run

LoadAndRemoveTextures("/folderWithManyImages", 2000)

in the console window.

imageMemoryProblem-WorksWithStableCycles.blend
imageMemoryProblem2pt80.blend

Code:

import datetime
import sys

import bpy
import bmesh
from mathutils import Vector
from mathutils import Euler
from bpy_extras.object_utils import world_to_camera_view
import random
import math
import json

import time
import os
import gc

def RemoveImageFromMemory (passedName):
    # Extra test because this can crash Blender.
    img = bpy.data.images[passedName]
    try:
        img.user_clear()
        can_continue = True
    except:
        can_continue = False
    
    if can_continue == True:
        try:
            bpy.data.images.remove(img, True)
            result = True
        except:
            result = False
    else:
        result = False
    return result

def CollectBackgroundImagePaths(sourceDirectoryPath, imageExtension):
    file_paths = []

    for root, dirs, files in os.walk(sourceDirectoryPath):
        for file_name in files:
            if file_name.endswith(imageExtension):
                file_paths.append(root + "/" + file_name)
    return file_paths

def SetBackgroundImage(backgroundImagePath):
    #Store reference to old image so it can be removed
    old_image_name = bpy.data.materials[0].node_tree.nodes.get("Image Texture").image.name

    #Load new image to be used
    new_image = bpy.data.images.load(filepath = backgroundImagePath)

    #Set the new image to be used
    bpy.data.materials[0].node_tree.nodes.get("Image Texture").image = new_image

    #Delete the old one from the file
    remove_succeeded = RemoveImageFromMemory(old_image_name)
    gc.collect()

    if not remove_succeeded:
        print("Error removing " + old_image_name)
    else:
        print("Removing " + old_image_name + " worked")

def LoadAndRemoveTextures(texturesFolderPath, numberToLoad):
    background_image_paths = CollectBackgroundImagePaths(texturesFolderPath, ".png")

    for example_index in range(0, numberToLoad):
        SetBackgroundImage(random.choice(background_image_paths))

        #If you alternate adding/removing images and rendering, it will eat all the ram on your system and crash
        bpy.ops.render.render( write_still=True )

**System Information** Operating system and graphics card Ubuntu 18.04, Nvidia GTX 960M **Blender Version** Broken: blender-2.80-17bc0567268-linux-glibc219-x86_64 **Working Blender Version** Blender 2.79b stable (ubuntu repository) with cycles **Short description of error** If you run the attached code and target it a directory with a large number of images, it will repeatedly: 1. Load an image from file 2. Set the image to be the content of a "Image Texture" node 3. Delete/remove the image that use to be the image texture node 4. Render As it processes images, blender takes more and more memory. Eventually it will crash. If you leave out step 4, then it will not. **Exact steps for others to reproduce the error** Based on a (as simple as possible) attached .blend file with minimum amount of steps Import python code, run LoadAndRemoveTextures("/folderWithManyImages", 2000) in the console window. [imageMemoryProblem-WorksWithStableCycles.blend](https://archive.blender.org/developer/F3919607/imageMemoryProblem-WorksWithStableCycles.blend) [imageMemoryProblem2pt80.blend](https://archive.blender.org/developer/F3919613/imageMemoryProblem2pt80.blend) Code: ``` import datetime import sys import bpy import bmesh from mathutils import Vector from mathutils import Euler from bpy_extras.object_utils import world_to_camera_view import random import math import json import time import os import gc def RemoveImageFromMemory (passedName): # Extra test because this can crash Blender. img = bpy.data.images[passedName] try: img.user_clear() can_continue = True except: can_continue = False if can_continue == True: try: bpy.data.images.remove(img, True) result = True except: result = False else: result = False return result def CollectBackgroundImagePaths(sourceDirectoryPath, imageExtension): file_paths = [] for root, dirs, files in os.walk(sourceDirectoryPath): for file_name in files: if file_name.endswith(imageExtension): file_paths.append(root + "/" + file_name) return file_paths def SetBackgroundImage(backgroundImagePath): #Store reference to old image so it can be removed old_image_name = bpy.data.materials[0].node_tree.nodes.get("Image Texture").image.name #Load new image to be used new_image = bpy.data.images.load(filepath = backgroundImagePath) #Set the new image to be used bpy.data.materials[0].node_tree.nodes.get("Image Texture").image = new_image #Delete the old one from the file remove_succeeded = RemoveImageFromMemory(old_image_name) gc.collect() if not remove_succeeded: print("Error removing " + old_image_name) else: print("Removing " + old_image_name + " worked") def LoadAndRemoveTextures(texturesFolderPath, numberToLoad): background_image_paths = CollectBackgroundImagePaths(texturesFolderPath, ".png") for example_index in range(0, numberToLoad): SetBackgroundImage(random.choice(background_image_paths)) #If you alternate adding/removing images and rendering, it will eat all the ram on your system and crash bpy.ops.render.render( write_still=True ) ```

Charles West commented

2018-07-12 21:06:54 +02:00

Added subscriber: @charlesrwest

Charles West commented

2018-07-13 16:29:21 +02:00

Further investigation reveals that switching the renderer from Eevee to cycles also makes it stop eating all the ram, so it seems to be Eevee specific.

Bastien Montagne commented

2018-07-25 16:00:34 +02:00

Added subscriber: @mont29

Bastien Montagne commented

2018-07-25 16:00:34 +02:00

Hrmmpfs… this is crashing here, in the scene_update call from render operator… depsgraph tries to access freed image ID, which is weird since relations are tagged to be updated in ID relink code.

No time to investigate further, but there are definitively issues here!

Hrmmpfs… this is crashing here, in the scene_update call from render operator… depsgraph tries to access freed image ID, which is weird since relations are tagged to be updated in ID relink code. No time to investigate further, but there are definitively issues here!

Charles West commented

2018-07-25 16:10:15 +02:00

If I may ask, is there anything I can do to bump up priority on this? I'm doing some deep learning research that requires 3 million+ examples to be generated and switching from Eevee to Cycles with low samples has made rendering time go from 2.5 days to something like 10 days while reducing quality.

Can I post a bounty or something?

If I may ask, is there anything I can do to bump up priority on this? I'm doing some deep learning research that requires 3 million+ examples to be generated and switching from Eevee to Cycles with low samples has made rendering time go from 2.5 days to something like 10 days while reducing quality. Can I post a bounty or something?

Zak Tab commented

2019-04-09 19:55:46 +02:00

Added subscriber: @zakleap

Zak Tab commented

2019-04-09 19:55:49 +02:00

Removed subscriber: @zakleap

Zak Tab commented

2019-04-09 19:55:51 +02:00

Added subscriber: @zakleap

Zak Tab commented

2019-04-09 19:59:14 +02:00

I have found that this causes the Syncing memory to keep going up unbounded. On first render, memory peak is around 4000Mb, on the second it goes up to 8000Mb. This has completely killed my ability to render my scene.

I have found that this causes the `Syncing` memory to keep going up unbounded. On first render, memory peak is around 4000Mb, on the second it goes up to 8000Mb. This has completely killed my ability to render my scene.

Philipp Oeser changed title from ~~Add/removing textures between renders causes memory leak/ram crash~~ to Python/BMesh: Add/removing textures between renders causes memory leak/ram crash

2019-11-05 12:32:50 +01:00

Greg Izatt commented

2019-12-18 21:08:32 +01:00

Added subscriber: @gizatt

Greg Izatt commented

2019-12-18 21:08:34 +01:00

Removed subscriber: @gizatt

Greg Izatt commented

2019-12-18 21:08:36 +01:00

Added subscriber: @gizatt

Richard Antalik commented

2020-02-10 22:26:35 +01:00

Added subscriber: @iss

Richard Antalik commented

2020-02-10 22:26:35 +01:00

Same "leak" happens with cycles as soon as I unpack list of loaded images in image editor (slowly and progressively - loading thumbnails?)
When I reload blend file images are gone, so they didn't have users.

Same "leak" happens with cycles as soon as I unpack list of loaded images in image editor (slowly and progressively - loading thumbnails?) When I reload blend file images are gone, so they didn't have users.

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Python/BMesh: Add/removing textures between renders causes memory leak/ram crash #55954