BLI_mmap not thread-safe on Linux/macOS (unfreed memory generating thumbnails) #91212

New Issue

Campbell Barton · 2021-09-06T12:20:59+02:00

Campbell Barton commented

2021-09-06 12:20:59 +02:00

Recently I noticed generating blend file thumbnails sometimes leaks memory, this is caused by 0f2ae614a1, which uses a linked list for error_handler_data.open_mmaps, threads are used to open blend file thumbnails which add/remove items without any locking.

Testing should be straightforward.

Remove thumbnail cache, (e.g. rm -rf ~/.cache/thumbnails)
Open a directory containing many blend files.
Wait for the thumbnails to load.
Close blender.

On exit ASAN reports this error (below).

Checking BLI_mmap it looks like all list access would need to locking, including handling errors.

Direct leak of 32 byte(s) in 1 object(s) allocated from:
    - 0 0x7f7ed1d97459 in __interceptor_calloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
    - 1 0x55cd8a26f9ac in MEM_lockfree_callocN /src/blender/intern/guardedalloc/intern/mallocn_lockfree_impl.c:236
    - 2 0x55cd8a00563a in BLI_genericNodeN /src/blender/source/blender/blenlib/intern/listbase.c:932
    - 3 0x55cd89eded19 in sigbus_handler_add /src/blender/source/blender/blenlib/intern/BLI_mmap.c:136
    - 4 0x55cd89edef8c in BLI_mmap_open /src/blender/source/blender/blenlib/intern/BLI_mmap.c:188
    - 5 0x55cd89fa181d in BLI_filereader_new_mmap /src/blender/source/blender/blenlib/intern/filereader_memory.c:130
    - 6 0x55cd83c2f97d in blo_filedata_from_file_descriptor /src/blender/source/blender/blenloader/intern/readfile.c:1223
    - 7 0x55cd83c2fec8 in blo_filedata_from_file_open /src/blender/source/blender/blenloader/intern/readfile.c:1270
    - 8 0x55cd83c30137 in blo_filedata_from_file_minimal /src/blender/source/blender/blenloader/intern/readfile.c:1293
    - 9 0x55cd83c31e31 in BLO_thumbnail_from_file /src/blender/source/blender/blenloader/intern/readfile.c:1513
    - 10 0x55cd84e68067 in imb_thumb_load_from_blendfile /src/blender/source/blender/imbuf/intern/thumbs_blend.c:72
    - 11 0x55cd84e6810b in IMB_thumb_load_blend /src/blender/source/blender/imbuf/intern/thumbs_blend.c:86
    - 12 0x55cd84e6540c in thumb_create_ex /src/blender/source/blender/imbuf/intern/thumbs.c:390
    - 13 0x55cd84e6676c in thumb_create_or_fail /src/blender/source/blender/imbuf/intern/thumbs.c:495
    - 14 0x55cd84e677e9 in IMB_thumb_manage /src/blender/source/blender/imbuf/intern/thumbs.c:656
    - 15 0x55cd893c5c44 in filelist_cache_preview_runf /src/blender/source/blender/editors/space_file/filelist.c:1472
    - 16 0x55cd8a24591e in Task::operator()() const /src/blender/source/blender/blenlib/intern/task_pool.cc:178
    - 17 0x55cd8a24a288 in tbb::internal::function_task<Task>::execute() /usr/include/tbb/task.h:1059
    #18 0x55cd83a89424 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop(tbb::internal::context_guard_helper<false>&, tbb::task*, long) (/src/cmake_debug/bin/blender+0x12db9424)

Recently I noticed generating blend file thumbnails sometimes leaks memory, this is caused by 0f2ae614a1, which uses a linked list for `error_handler_data.open_mmaps`, threads are used to open blend file thumbnails which add/remove items without any locking. Testing should be straightforward. - Remove thumbnail cache, (e.g. `rm -rf ~/.cache/thumbnails`) - Open a directory containing many blend files. - Wait for the thumbnails to load. - Close blender. On exit ASAN reports this error (below). Checking `BLI_mmap` it looks like all list access would need to locking, including handling errors. ---- ``` Direct leak of 32 byte(s) in 1 object(s) allocated from: - 0 0x7f7ed1d97459 in __interceptor_calloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154 - 1 0x55cd8a26f9ac in MEM_lockfree_callocN /src/blender/intern/guardedalloc/intern/mallocn_lockfree_impl.c:236 - 2 0x55cd8a00563a in BLI_genericNodeN /src/blender/source/blender/blenlib/intern/listbase.c:932 - 3 0x55cd89eded19 in sigbus_handler_add /src/blender/source/blender/blenlib/intern/BLI_mmap.c:136 - 4 0x55cd89edef8c in BLI_mmap_open /src/blender/source/blender/blenlib/intern/BLI_mmap.c:188 - 5 0x55cd89fa181d in BLI_filereader_new_mmap /src/blender/source/blender/blenlib/intern/filereader_memory.c:130 - 6 0x55cd83c2f97d in blo_filedata_from_file_descriptor /src/blender/source/blender/blenloader/intern/readfile.c:1223 - 7 0x55cd83c2fec8 in blo_filedata_from_file_open /src/blender/source/blender/blenloader/intern/readfile.c:1270 - 8 0x55cd83c30137 in blo_filedata_from_file_minimal /src/blender/source/blender/blenloader/intern/readfile.c:1293 - 9 0x55cd83c31e31 in BLO_thumbnail_from_file /src/blender/source/blender/blenloader/intern/readfile.c:1513 - 10 0x55cd84e68067 in imb_thumb_load_from_blendfile /src/blender/source/blender/imbuf/intern/thumbs_blend.c:72 - 11 0x55cd84e6810b in IMB_thumb_load_blend /src/blender/source/blender/imbuf/intern/thumbs_blend.c:86 - 12 0x55cd84e6540c in thumb_create_ex /src/blender/source/blender/imbuf/intern/thumbs.c:390 - 13 0x55cd84e6676c in thumb_create_or_fail /src/blender/source/blender/imbuf/intern/thumbs.c:495 - 14 0x55cd84e677e9 in IMB_thumb_manage /src/blender/source/blender/imbuf/intern/thumbs.c:656 - 15 0x55cd893c5c44 in filelist_cache_preview_runf /src/blender/source/blender/editors/space_file/filelist.c:1472 - 16 0x55cd8a24591e in Task::operator()() const /src/blender/source/blender/blenlib/intern/task_pool.cc:178 - 17 0x55cd8a24a288 in tbb::internal::function_task<Task>::execute() /usr/include/tbb/task.h:1059 #18 0x55cd83a89424 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop(tbb::internal::context_guard_helper<false>&, tbb::task*, long) (/src/cmake_debug/bin/blender+0x12db9424) ```

Lukas Stockner was assigned by Campbell Barton

2021-09-06 12:20:59 +02:00

Campbell Barton commented

2021-09-06 12:20:59 +02:00

Added subscribers: @ideasman42, @mont29, @brecht

Campbell Barton changed title from ~~BLI_mmap not thread-safe (unfreed memory generating thumbnails)~~ to BLI_mmap not thread-safe on Linux/macOS (unfreed memory generating thumbnails)

2021-09-06 12:21:21 +02:00

Germano Cavalcante commented

2021-10-01 16:47:34 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

BLI_mmap not thread-safe on Linux/macOS (unfreed memory generating thumbnails) #91212