Joining objects can create more customdata layers (e.g. UVMaps) than allowed, can eat huge amounts of RAM, bloats the filesize #111608

New Issue

Philipp Oeser · 2023-08-28T13:30:46+02:00

Philipp Oeser commented

2023-08-28 13:30:46 +02:00

System Information
Operating system: Linux-6.1.18-200.fc37.x86_64-x86_64-with-glibc2.36 64 Bits, WAYLAND UI
Graphics card: AMD Radeon Graphics (renoir, LLVM 15.0.7, DRM 3.49, 6.1.18-200.fc37.x86_64) AMD 4.6 (Core Profile) Mesa 22.3.7

Blender Version
Broken: version: 4.0.0 Alpha, branch: main, commit date: 2023-08-27 18:51, hash: 8a735d0aa4c7
Worked: never

Short description of error
Joining objects can create more customdata layers (e.g. UVMaps) than allowed.
While there is no hard limit for UV layers (yet), we usually restrict this to 8 from the UI.
Why can this be a problem?

See reports #111564, #71823, #96926

Joining many (hundreds or thousands -- which is not uncommon for CAD data) small objects with uniquely named UV layers will result in many, many UV layers (not only UVs, other customdata layers / attributes may be affected as well, but encountered most frequently for those), each then taking up memory/space for the whole -- now possibly very large -- mesh.

This will - unexpectedly - bloat up the filesize tremendously, also take up huge amounts of RAM for the joining operation, leading to possible crashes.
This is also not actually supported for UVMaps (the UI limit/restriction of 8 is there for a reason since e.g. Eevee, probably Cycles, too, dont handle more then 8).

Exact steps for others to reproduce the error

open uv_maps_on_merge_single_user_mesh_unique_uv_layer_names.blend (1.6MB)
join all objects, save (file is now 3.8MB, the joined object has 121 UVmaps)
the problem gets much worse if the resulting object is larger and/or if object count increases

**System Information** Operating system: Linux-6.1.18-200.fc37.x86_64-x86_64-with-glibc2.36 64 Bits, WAYLAND UI Graphics card: AMD Radeon Graphics (renoir, LLVM 15.0.7, DRM 3.49, 6.1.18-200.fc37.x86_64) AMD 4.6 (Core Profile) Mesa 22.3.7 **Blender Version** Broken: version: 4.0.0 Alpha, branch: main, commit date: 2023-08-27 18:51, hash: `8a735d0aa4c7` Worked: never **Short description of error** Joining objects can create more customdata layers (e.g. UVMaps) than allowed. While there is no hard limit for UV layers (yet), we usually restrict this to 8 from the UI. Why can this be a problem? See reports #111564, #71823, #96926 Joining many (hundreds or thousands -- which is not uncommon for CAD data) small objects with uniquely named UV layers will result in many, many UV layers (not only UVs, other customdata layers / attributes may be affected as well, but encountered most frequently for those), each then taking up memory/space for the whole -- now possibly very large -- mesh. This will - unexpectedly - bloat up the filesize tremendously, also take up huge amounts of RAM for the joining operation, leading to possible crashes. This is also not actually supported for UVMaps (the UI limit/restriction of 8 is there for a reason since e.g. Eevee, probably Cycles, too, dont handle more then 8). **Exact steps for others to reproduce the error** - open `uv_maps_on_merge_single_user_mesh_unique_uv_layer_names.blend` (1.6MB) - join all objects, save (file is now 3.8MB, the joined object has 121 UVmaps) - the problem gets much worse if the resulting object is larger and/or if object count increases

uv_maps_on_merge_single_user_mesh_unique_uv_layer_names.blend

1.5 MiB

Philipp Oeser added the

labels 2023-08-28 13:30:46 +02:00

Philipp Oeser commented

2023-08-28 13:31:50 +02:00

Will put on my desk for further investigation, possible solutions:

either limiting the amount of UV layers upon joining to 8 (as we elsewhere in the UI)
merging the UV layers based upon index (so merge all first uv layers of all sources into one on the destination, merge all second uv layers of all sources into a second uv layer on destination, ...) -- this could be an option on the operator
if the above doesnt work out for some reason, at least spit out a warning in the UI that more than 8 UV layers have been created

Will put on my desk for further investigation, possible solutions: - either limiting the amount of UV layers upon joining to 8 (as we elsewhere in the UI) - merging the UV layers based upon index (so merge all first uv layers of all sources into one on the destination, merge all second uv layers of all sources into a second uv layer on destination, ...) -- this could be an option on the operator - if the above doesnt work out for some reason, at least spit out a warning in the UI that more than 8 UV layers have been created

Philipp Oeser commented

2023-08-28 13:34:31 +02:00

Note to myself: LayerTypeInfo could have a layers_max() function for CD_PROP_FLOAT2 (it did have one for CD_MTFACE -- layerMaxNum_tface)

Note to myself: `LayerTypeInfo` could have a `layers_max()` function for `CD_PROP_FLOAT2` (it did have one for `CD_MTFACE` -- `layerMaxNum_tface`)

Philipp Oeser added

and removed

labels 2023-08-28 13:34:53 +02:00

Philipp Oeser self-assigned this 2023-08-28 13:34:57 +02:00

Philipp Oeser referenced this issue

2023-08-28 13:35:24 +02:00

Merging/joining into single object massively increases file size / Fills all available memory. #111564

Philipp Oeser referenced this issue

2023-08-28 13:35:40 +02:00

Merging/joining into single object can massively increase file size #96926

Philipp Oeser referenced this issue

2023-08-28 13:35:56 +02:00

Merging objects that use meshes with users count of 1 increases the file size massively #71823

Philipp Oeser referenced this issue

2023-08-28 13:54:46 +02:00

Customdata merging does not check layer limit on destination #111609

Philipp Oeser commented

2023-08-28 14:05:13 +02:00

@Baardaap might be interested

Philipp Oeser referenced this issue

2023-08-28 14:37:22 +02:00

Customdata merging does not check layer limit on destination #111609

Philipp Oeser commented

2023-08-28 15:00:58 +02:00

Just dropping a workaround for folks running into this: you can of course rename the UVs (probably best thing to do is by index) prior to joining (so names are equal on participating meshes -- this way the layers get merged instead of created)

YimingWu commented

2023-08-28 15:12:17 +02:00

If I'm understanding this correctly, this behaviour is because that every face corner should have a UV coordinate in all uv layers present in the mesh? (Or every layer of UV will have all face corners of the mesh, I guess the same idea)

If I'm understanding this correctly, this behaviour is because that every face corner should have a UV coordinate in _all_ uv layers present in the mesh? (Or every layer of UV will have all face corners of the mesh, I guess the same idea)

Philipp Oeser commented

2023-08-28 15:20:01 +02:00

Yes, all layers have the size of the whole mesh.

Quoting @HooglyBoogly from chat:

It's theoretically possible to optimize attribute storage for cases like this, to avoid keeping the giant array of zeros

Yes, all layers have the size of the whole mesh. Quoting @HooglyBoogly from chat: >It's theoretically possible to optimize attribute storage for cases like this, to avoid keeping the giant array of zeros

👍 1

Hans Goudey commented

2023-08-28 15:30:37 +02:00

I should clarify we're pretty far from being able to do that (we have to replaces uses of CustomData_* with the attribute API first), but it's good to keep such sparse attribute storage in mind.

I do think it's better if joining stays predictable/simple and name based. A "Rename UV Maps by Index" operator might make sense, though it may be a bit specific?

I should clarify we're pretty far from being able to do that (we have to replaces uses of `CustomData_*` with the attribute API first), but it's good to keep such sparse attribute storage in mind. I do think it's better if joining stays predictable/simple and name based. A "Rename UV Maps by Index" operator might make sense, though it may be a bit specific?

👀 1

Hans Goudey commented

2023-08-28 15:31:29 +02:00

I also want to clarify that there is no limit to the number of UV attributes in the design now-- that was an important part of the switch to the generic attribute API.

Philipp Oeser commented

2023-08-28 15:37:59 +02:00

I also want to clarify that there is no limit to the number of UV attributes in the design now-- that was an important part of the switch to the generic attribute API.

We still limit it from the UI though (at least from the UIList, adding more than 8 is prevented)

> I also want to clarify that there is no limit to the number of UV attributes in the design now-- that was an important part of the switch to the generic attribute API. We still limit it from the UI though (at least from the UIList, adding more than 8 is prevented)

Hans Goudey commented

2023-08-28 15:40:22 +02:00

We still limit it from the UI though (at least from the UIList, adding more than 8 is prevented)

Interesting, didn't realize that. I know the transition to remove the limit from some areas is still incomplete, maybe that's the reasoning (hence the "on a design level")

> We still limit it from the UI though (at least from the UIList, adding more than 8 is prevented) Interesting, didn't realize that. I know the transition to remove the limit from some areas is still incomplete, maybe that's the reasoning (hence the "on a design level")

Philipp Oeser commented

2023-08-29 13:55:59 +02:00

We still limit it from the UI though (at least from the UIList, adding more than 8 is prevented)

Interesting, didn't realize that. I know the transition to remove the limit from some areas is still incomplete, maybe that's the reasoning (hence the "on a design level")

The limit is there for a reason, see BKE_mesh_validate_all_customdata

More UV layers than allowed, last ones won't be available for render, shaders, etc.

The limit cant be removed from everywhere (at least not in a way that layers above the limit will actually work as expected), so even if we would allow them internally (which might not be the best idea due to the problems presented in this report), we have to counter-act on the UX level (warnings, cleanup operators, defaulting to merge by index in the join operator, ...)

> > We still limit it from the UI though (at least from the UIList, adding more than 8 is prevented) > > Interesting, didn't realize that. I know the transition to remove the limit from some areas is still incomplete, maybe that's the reasoning (hence the "on a design level") > The limit is there for a reason, see `BKE_mesh_validate_all_customdata` >More UV layers than allowed, last ones won't be available for render, shaders, etc. The limit cant be removed from everywhere (at least not in a way that layers above the limit will actually work as expected), so even if we would allow them internally (which might not be the best idea due to the problems presented in this report), we have to counter-act on the UX level (warnings, cleanup operators, defaulting to merge by index in the join operator, ...)

Philipp Oeser referenced this issue from a commit

2023-08-29 18:02:44 +02:00

Fix: Customdata merging does not check layer limit on destination

Philipp Oeser referenced this issue from a commit

2023-09-12 17:37:20 +02:00

Fix: Customdata merging does not check layer limit on destination

Martijn Versteegh commented

2023-09-25 20:34:46 +02:00

More UV layers than allowed, last ones won't be available for render, shaders, etc.

I'm not completely sure about this, since it has been a while I really dove into this code, but I think only the uv layers referenced from the material get copied to the renderer. So that gives it some leeway as a mesh can have more uv layers.

Extra UV layers can be useful in GN trees which have no problem using them. But it does lead to a quite messy situation which would need clear warnings in the UI some maybe it would be simpler to just keep the restriction on the number of UV layers in place.

> More UV layers than allowed, last ones won't be available for render, shaders, etc. I'm not completely sure about this, since it has been a while I really dove into this code, but I _think_ only the uv layers referenced from the material get copied to the renderer. So that gives it some leeway as a mesh can have more uv layers. Extra UV layers can be useful in GN trees which have no problem using them. But it does lead to a quite messy situation which would need clear warnings in the UI some maybe it would be simpler to just keep the restriction on the number of UV layers in place.

Philipp Oeser commented

2024-03-27 14:10:22 +01:00

Stepping down since I currently not plan on actually working on this (although it is still a big issue)

Philipp Oeser removed their assignment 2024-03-27 14:10:32 +01:00

Philipp Oeser referenced this issue

2024-06-05 10:44:21 +02:00

Joining objects deletes UV map #64245

Bart van der Braak added

Type

Bug

and removed

Type