Merging objects that use meshes with users count of 1 increases the file size massively #71823

Closed
opened 2019-11-23 16:14:44 +01:00 by roxlu - · 14 comments

System Information
Operating system: Linux-5.3.12-arch1-1-x86_64-with-arch 64 Bits
Graphics card: GeForce GTX 980/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 440.31

Blender Version
Broken: version: 2.81 (sub 16), branch: master, commit date: 2019-11-20 14:27, hash: 26bd5ebd42
Worked: (optional)

Short description of error
I'm working on a architectural model. I imported the model from Rhino 3D using Collada. The file has about 300K vertices. I'm now merging similar objects by selecting them and pressing CTRL + J. After merging some objects the file size increased from 82MB to 1.4GB. First I thought that this was caused because I was merging objects that were sharing the same mesh data, but when I iterate over the meshes and print mesh.users their value is all 1, for each mesh. I purged all ophaned data too.

As a test, I exported the .blend file to .obj.

  • The resulting .obj file was ~35MB.
  • Then I imported the file in a new .blend and saved that one, the file size of this .blend is 57MB.

I've started a discussion here too: https://blenderartists.org/t/massive-file-size-increase-after-joining-meshes-objects/1192317/4

UPDATE 1
I toggled the - [x] compression box when saving and now the file is 18MB. I still think this is a bug as 300K vertices does not result in such a file size increase. Opening that file of 18MB results in ~2.6GB memory usage.
https://imgur.com/a/22hNroI

UPDATE 2
I think this is partly caused by the undo buffer somehow or at least, this issue is twofold; disabling the undo buffer allows me at least to continue working.

What can cause this increase?

Exact steps for others to reproduce the error
I'm using a private 3D model which I can't share. I'm happy to share this privately but I can't post it online.

**System Information** Operating system: Linux-5.3.12-arch1-1-x86_64-with-arch 64 Bits Graphics card: GeForce GTX 980/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 440.31 **Blender Version** Broken: version: 2.81 (sub 16), branch: master, commit date: 2019-11-20 14:27, hash: `26bd5ebd42` Worked: (optional) **Short description of error** I'm working on a architectural model. I imported the model from Rhino 3D using Collada. The file has about 300K vertices. I'm now merging similar objects by selecting them and pressing CTRL + J. After merging some objects the file size increased from 82MB to 1.4GB. First I thought that this was caused because I was merging objects that were sharing the same mesh data, but when I iterate over the meshes and print `mesh.users` their value is all 1, for each mesh. I purged all ophaned data too. As a test, I exported the .blend file to .obj. - The resulting .obj file was ~35MB. - Then I imported the file in a new .blend and saved that one, the file size of this .blend is 57MB. I've started a discussion here too: https://blenderartists.org/t/massive-file-size-increase-after-joining-meshes-objects/1192317/4 **UPDATE 1** I toggled the `- [x] compression` box when saving and now the file is 18MB. I still think this is a bug as 300K vertices does not result in such a file size increase. Opening that file of 18MB results in ~2.6GB memory usage. https://imgur.com/a/22hNroI **UPDATE 2** I think this is partly caused by the undo buffer somehow or at least, this issue is twofold; disabling the undo buffer allows me at least to continue working. What can cause this increase? **Exact steps for others to reproduce the error** I'm using a private 3D model which I can't share. I'm happy to share this privately but I can't post it online.
Author

Added subscriber: @roxlu

Added subscriber: @roxlu

Added subscriber: @AnthonyEdlin

Added subscriber: @AnthonyEdlin

Yes if you do a series of object joins it's creating a massive undo stack, storing the data multiple times. You can do like you did and disable undo, also if you do not join them 1 by 1 but as one big group it will only be 1 undo. I don't think there is a bug here.

Yes if you do a series of object joins it's creating a massive undo stack, storing the data multiple times. You can do like you did and disable undo, also if you do not join them 1 by 1 but as one big group it will only be 1 undo. I don't think there is a bug here.
Author

Ok, good to know! The crashes were indeed most likely be caused because my PC ran out of mem. But the file size shouldn't change from 18MB to 1.25GB.

Ok, good to know! The crashes were indeed most likely be caused because my PC ran out of mem. But the file size shouldn't change from 18MB to 1.25GB.
Author

I just noticed, a day later (?!) that when I open the file my memory goes from 18MB to 466MB which is still al lot but understandable. I removed my /tmp files and I'm not sure if blender uses shared memory, but I think something went wrong with cache files. But that's just a wild guess.

I just noticed, a day later (?!) that when I open the file my memory goes from 18MB to 466MB which is still al lot but understandable. I removed my /tmp files and I'm not sure if blender uses shared memory, but I think something went wrong with cache files. But that's just a wild guess.

Changed status from 'Open' to: 'Archived'

Changed status from 'Open' to: 'Archived'
Campbell Barton self-assigned this 2019-11-28 08:41:29 +01:00

Changed status from 'Archived' to: 'Needs Developer To Reproduce'

Changed status from 'Archived' to: 'Needs Developer To Reproduce'

Re-opened because closing should always give a response for why.

Re-opened because closing should always give a response for why.
Member

Added subscriber: @lichtwerk

Added subscriber: @lichtwerk
Member

OK, we now have #96926 (Merging/joining into single object can massively increase file size) which seems similar and are trying to come up with a repro case there.

So, the reason this was probably closed here is that we dont really have such a repro case, @roxlu : if you can provide one, please share it in #96926 instead.
Will close this one again.

OK, we now have #96926 (Merging/joining into single object can massively increase file size) which seems similar and are trying to come up with a repro case there. So, the reason this was probably closed here is that we dont really have such a repro case, @roxlu : if you can provide one, please share it in #96926 instead. Will close this one again.
Member

Closed as duplicate of #96926

Closed as duplicate of #96926

Morning, First post ! I am currently trying to move our teams process away from Maya to Blender, we deal with a massive amount of CAD data for numerous car manufactures, we are experiencing exactly the same issue as above. I've screen captured a simplified version of our workflow to see if it is a bug or if I'm doing something stupid. I'll outline the steps.

Blender Version 3.6.2
Undo's disabled.
Import 1.02 FBX file.
Unparent parts from empties and apply all transforms. ( No increase in memory usage. )
Join similar textured meshes, as more meshes are joined memory usage increases ( in some cases on larger files filling the entire system memory )
Purge all orphaned data and save.
As you can see just these simple operation increase the file size from 2.11gb to 46.7gb, I've tried everything I can think of to find the leak/unused data but I am at a loss.
Reopen Save and as you can see 46.7gb of system memory used but only 2.11gb is my data, this is where it gets odd and I believe Blender is saving a ton of orphaned data or just garbage ( Remember undo's are disabled and I use Clean-ups constantly. )
Export to an obj and the file drops back down to 2.48gb, reopen in blender and all the data is the same and nothing is missing but the file is back to 2.04gb again ! madness. The export and reimport flushes all the unwanted data but is a pain to keep doing. any help would be appreciated. Link below to the screen capture, couldn't get it smaller than 1.2gb at 4k Sorry.

https://1drv.ms/v/s!AoarOZXHLMNWirJ5XDDjIchwGRp2VA?e=fWChO1

A link to a zip below as Onedrive only seems to stream in 1080p

https://1drv.ms/u/s!AoarOZXHLMNWirJ6OKSxqjOvCXjFCg?e=IyhFk2

Good luck.

Morning, First post ! I am currently trying to move our teams process away from Maya to Blender, we deal with a massive amount of CAD data for numerous car manufactures, we are experiencing exactly the same issue as above. I've screen captured a simplified version of our workflow to see if it is a bug or if I'm doing something stupid. I'll outline the steps. Blender Version 3.6.2 Undo's disabled. Import 1.02 FBX file. Unparent parts from empties and apply all transforms. ( No increase in memory usage. ) Join similar textured meshes, as more meshes are joined memory usage increases ( in some cases on larger files filling the entire system memory ) Purge all orphaned data and save. As you can see just these simple operation increase the file size from 2.11gb to 46.7gb, I've tried everything I can think of to find the leak/unused data but I am at a loss. Reopen Save and as you can see 46.7gb of system memory used but only 2.11gb is my data, this is where it gets odd and I believe Blender is saving a ton of orphaned data or just garbage ( Remember undo's are disabled and I use Clean-ups constantly. ) Export to an obj and the file drops back down to 2.48gb, reopen in blender and all the data is the same and nothing is missing but the file is back to 2.04gb again ! madness. The export and reimport flushes all the unwanted data but is a pain to keep doing. any help would be appreciated. Link below to the screen capture, couldn't get it smaller than 1.2gb at 4k Sorry. https://1drv.ms/v/s!AoarOZXHLMNWirJ5XDDjIchwGRp2VA?e=fWChO1 A link to a zip below as Onedrive only seems to stream in 1080p https://1drv.ms/u/s!AoarOZXHLMNWirJ6OKSxqjOvCXjFCg?e=IyhFk2 Good luck.
Member

So this was reported again in #111564

And the reason seems to be that joining does not limit the number of UV layers to 8 (as done elsewhere in blender).
So you might end up with hundreds or thousands of UV layers (and each of them occupying memory for a -- now very large -- mesh).

I think this is probably what was the issue in #96926 as well.

I think the situation should still be improved:

  • either limiting the amount of UV layers upon joining to 8 (as we elsewhere in the UI)
  • merging the UV layers based upon index (so merge all first uv layers of all sources into one on the destination, merge all second uv layers of all sources into a second uv layer on destination, ...) -- this could be an option on the operator
  • if the above doesnt work out for some reason, at least spit out a warning in the UI that more than 8 UV layers have been created

I will open another issue for this shortly...

So this was reported again in #111564 And the reason seems to be that joining does not limit the number of UV layers to 8 (as done elsewhere in blender). So you might end up with hundreds or thousands of UV layers (and each of them occupying memory for a -- now very large -- mesh). I think this is probably what was the issue in #96926 as well. I think the situation should still be improved: - either limiting the amount of UV layers upon joining to 8 (as we elsewhere in the UI) - merging the UV layers based upon index (so merge all first uv layers of all sources into one on the destination, merge all second uv layers of all sources into a second uv layer on destination, ...) -- this could be an option on the operator - if the above doesnt work out for some reason, at least spit out a warning in the UI that more than 8 UV layers have been created I will open another issue for this shortly...
Member

I opened #111608 for this.

I opened #111608 for this.
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset System
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Code Documentation
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
FBX
Interest
Freestyle
Interest
Geometry Nodes
Interest
glTF
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Viewport & EEVEE
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Asset Browser Project
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Asset System
Module
Core
Module
Development Management
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Module
Viewport & EEVEE
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Severity
High
Severity
Low
Severity
Normal
Severity
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
5 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#71823
No description provided.