Importing Alembic fails after more than ~510 individual cache files #105081

New Issue

Stefan Kreller · 2023-02-22T17:34:06+01:00

Stefan Kreller commented

2023-02-22 17:34:06 +01:00

System Information
Operating system: Windows 10
Graphics card: RTX 2080

Blender Version
Broken: 3.4.1, 3.5.0
Worked: -

Short description of error
I am trying to import a lot of alembic character files. After ~510 successful caches following imports fail with error Could not open Alembic archive for reading! See console for detail. The console additionally prints Could not open as ogawa file from provided streams Individually all of my alembic files import without problems. The alembics are exported from blender. Importing fails for complex characters, as well as the default cube exported as alembic.

Exact steps for others to reproduce the error
Open the attached blend file and run the script from the text editor. It will start to import the (attached) cube.abc without problems until it gets to roughly the 510th import.
At this point it is not possible to import any more alembic caches.

**System Information** Operating system: Windows 10 Graphics card: RTX 2080 **Blender Version** Broken: 3.4.1, 3.5.0 Worked: - **Short description of error** I am trying to import a lot of alembic character files. After ~510 successful caches following imports fail with error `Could not open Alembic archive for reading! See console for detail.` The console additionally prints `Could not open as ogawa file from provided streams` Individually all of my alembic files import without problems. The alembics are exported from blender. Importing fails for complex characters, as well as the default cube exported as alembic. **Exact steps for others to reproduce the error** Open the attached blend file and run the script from the text editor. It will start to import the (attached) `cube.abc` without problems until it gets to roughly the 510th import. At this point it is not possible to import any more alembic caches.

AlembicImportBug.zip

101 KiB

Stefan Kreller added the

labels 2023-02-22 17:34:07 +01:00

Jesse Yurkovich commented

2023-02-23 00:05:54 +01:00

I can confirm on both Windows and Linux although the counts differ. Both are affected by limits enforced by the OS on the number of open files for a given process. Linux is higher at around 1024 or so (ulimit -n), while Windows is 512 by default. The failure here occurs at 509 because stdout, stderr, and stdin count towards the limit and those are always open.

Increasing the limit on Windows is possible with a code change but only up to a maximum of 8192 on modern platforms. Linux is more difficult as it would require admin rights to increase I believe.

Ultimately this is caused because of the interaction between depsgraph and BKE_cachefile_eval. The deps graph requests the cachefile and for Alembic that means opening the file and keeping it open for, effectively, the entire time. The true fix would be to only require opening of the file when absolutely necessary and then closing soon after.

I can confirm on both Windows and Linux although the counts differ. Both are affected by limits enforced by the OS on the number of open files for a given process. Linux is higher at around 1024 or so (`ulimit -n`), while Windows is 512 by default. The failure here occurs at 509 because stdout, stderr, and stdin count towards the limit and those are always open. Increasing the limit on Windows is possible with a code change but only up to a maximum of 8192 on modern platforms. Linux is more difficult as it would require admin rights to increase I believe. Ultimately this is caused because of the interaction between depsgraph and `BKE_cachefile_eval`. The deps graph requests the cachefile and for Alembic that means opening the file and keeping it open for, effectively, the entire time. The true fix would be to only require opening of the file when absolutely necessary and then closing soon after.

Jesse Yurkovich added

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Importing Alembic fails after more than ~510 individual cache files #105081