Alembic split normal export issue #71130

New Issue

Andrew · 2019-10-27T11:41:48+01:00

Andrew commented

2019-10-27 11:41:48 +01:00

System Information
Operating system: Ubuntu 18.04
Graphics card: NV RTX 2060 super

Blender Version
Broken: 2.81, 892c3891ed, 2019-10-24 and 2.82-9aa1bb6388ac 10-23

Short description of error
Split normals do not seem to be exported properly from Blender.

Exact steps for others to reproduce the error

Open the attached badshelf.blend, turn on display of split normals and note that it looks like shelf-normals-blend-file.png:
badshelf.blend

The split normals are pointing in the expected direction -- the same as their corresponding face normal.
Export the shelf to an abc file with normal checkbox enabled in the alembic export settings.
Create a new scene in blender and import the abc file that you just exported.

Actual Result:

Firstly, note that the shading looks incorrect, as in shelf-normals-abc-file.png:

Also note that if you turn on split normal display, the split normals are pointing in a different direction than they were in the blend file. They seem to be using a smoothed normal instead of the corresponding face normal as expected.

Expected result:
After importing the abc file, the shading should look similar to the original badshelf.blend file. Upon turning on split normal display, the split normals should be pointing in the same direction as the split normals from the original blend file.

Possibly related issue:
https://developer.blender.org/T56792 (supposedly fixed)

**System Information** Operating system: Ubuntu 18.04 Graphics card: NV RTX 2060 super **Blender Version** Broken: 2.81, 892c3891ed0b, 2019-10-24 and 2.82-9aa1bb6388ac 10-23 **Short description of error** Split normals do not seem to be exported properly from Blender. **Exact steps for others to reproduce the error** 1. Open the attached badshelf.blend, turn on display of split normals and note that it looks like shelf-normals-blend-file.png: [badshelf.blend](https://archive.blender.org/developer/F7862235/badshelf.blend) ![shelf-normals-blend-file.png](https://archive.blender.org/developer/F7862231/shelf-normals-blend-file.png) The split normals are pointing in the expected direction -- the same as their corresponding face normal. 2. Export the shelf to an abc file with normal checkbox enabled in the alembic export settings. 3. Create a new scene in blender and import the abc file that you just exported. Actual Result: * Firstly, note that the shading looks incorrect, as in shelf-normals-abc-file.png: ![shelf-normals-abc-file.png](https://archive.blender.org/developer/F7862233/shelf-normals-abc-file.png) Also note that if you turn on split normal display, the split normals are pointing in a different direction than they were in the blend file. They seem to be using a smoothed normal instead of the corresponding face normal as expected. Expected result: After importing the abc file, the shading should look similar to the original badshelf.blend file. Upon turning on split normal display, the split normals should be pointing in the same direction as the split normals from the original blend file. Possibly related issue: https://developer.blender.org/T56792 (supposedly fixed)

Andrew commented

2019-10-27 11:41:48 +01:00

Added subscriber: @ajohnson223

Andrew commented

2019-10-27 11:42:55 +01:00

Added subscriber: @dr.sybren

Andrew commented

2019-10-27 11:42:55 +01:00

Also, calling out @dr.sybren since he appears to be the resident alembic dev...

Sybren A. Stüvel self-assigned this 2019-10-29 09:24:23 +01:00

Steffen Dünner commented

2019-10-29 12:21:41 +01:00

Added subscriber: @SteffenD

blender-admin commented

2019-10-29 16:51:55 +01:00

This issue was referenced by blender/documentation@5901

Sybren A. Stüvel commented

2019-10-29 16:52:31 +01:00

Changed status from 'Open' to: 'Resolved'

Sybren A. Stüvel closed this issue

2019-10-29 16:52:31 +01:00

Sybren A. Stüvel commented

2019-10-29 16:54:15 +01:00

This is working as designed; only when the 'Auto Smooth' checkbox is enabled, custom loop normals are written to the Alembic file. However, this wasn't described in the manual at all, so I made an addition to that to clarify this behaviour.

Andrew commented

2019-10-30 14:20:05 +01:00

@dr.sybren Thanks, but can there please be some additional clarification on this?

With fbx export, the process to export something such that the normals come back in properly is:

Open a blend file.
Export an fbx
New scene, Import fbx

Result: Normals match the original blend file

But it doesn't work this way with abc, and auto smooth doesn't seem to really resolve the issue from what I'm seeing so far. More concretely, if you do the following:

Go to https://www.blender.org/download/demo-files/
Download the Architectural Visualization scene.
Open the scene
Export scene as abc
Create a new scene then reimport the exported abc.

How do these steps need to change to achieve all the same normals that were in the original blend file? I've tried a number of things, among them:
Set auto smooth with angle 0 on all objects -- the shelves in the scene are correct, but anything that had some smooth normals of course come back in faceted. I'd somehow have to "guess" the right auto smooth value for every object in the scene prior to export, or something?

@dr.sybren Thanks, but can there please be some additional clarification on this? With fbx export, the process to export something such that the normals come back in properly is: 1. Open a blend file. 2. Export an fbx 3. New scene, Import fbx Result: Normals match the original blend file But it doesn't work this way with abc, and auto smooth doesn't seem to really resolve the issue from what I'm seeing so far. More concretely, if you do the following: 1. Go to https://www.blender.org/download/demo-files/ 2. Download the Architectural Visualization scene. 3. Open the scene 4. Export scene as abc 5. Create a new scene then reimport the exported abc. How do these steps need to change to achieve all the same normals that were in the original blend file? I've tried a number of things, among them: Set auto smooth with angle 0 on all objects -- the shelves in the scene are correct, but anything that had some smooth normals of course come back in faceted. I'd somehow have to "guess" the right auto smooth value for every object in the scene prior to export, or something?

Andrew commented

2019-10-30 14:36:52 +01:00

For posterity, the solution to preserve normals within an abc such that they match the blend file is to do the following prior to export:

for obj in bpy.data.objects:
    if obj.type == 'MESH':
        obj.data.use_auto_smooth = True
        if not obj.data.has_custom_normals:
            bpy.context.view_layer.objects.active = obj
            bpy.ops.mesh.customdata_custom_splitnormals_add()

It kind of feels to me like this should be a part of the exporter, at least behind a checkbox or something? The different exporters should have a consistent behavior with respect to "normals idempotence"... fbx seems to have this property whereas abc does not, unless you massage the scene yourself first by adding custom splitnormals layers on everything and turning on autosmooth. I thought that's what the "Normals" checkbox in the abc export option actually does...

For posterity, the solution to preserve normals within an abc such that they match the blend file is to do the following prior to export: ``` for obj in bpy.data.objects: if obj.type == 'MESH': obj.data.use_auto_smooth = True if not obj.data.has_custom_normals: bpy.context.view_layer.objects.active = obj bpy.ops.mesh.customdata_custom_splitnormals_add() ``` It kind of feels to me like this should be a part of the exporter, at least behind a checkbox or something? The different exporters should have a consistent behavior with respect to "normals idempotence"... fbx seems to have this property whereas abc does not, unless you massage the scene yourself first by adding custom splitnormals layers on everything and turning on autosmooth. I thought that's what the "Normals" checkbox in the abc export option actually does...

Sybren A. Stüvel commented

2019-10-30 16:31:06 +01:00

Changed status from 'Resolved' to: 'Open'

Sybren A. Stüvel reopened this issue

2019-10-30 16:31:06 +01:00

Sybren A. Stüvel commented

2019-10-30 16:31:06 +01:00

Added subscriber: @mont29

Sybren A. Stüvel commented

2019-10-30 16:31:06 +01:00

The "Normals" checkbox in the exporter options just enables or disables the exporting of normals. If it's disabled, normals are not written to the ABC file at all. When it's enabled, there are various ways to write the normals to Alembic; currently the ones in use are 'face-varying' and 'vertex-varying'. The thing is, we can only choose one or the other, so we either store loop normals ('face-varying') or vertex normals ('vertex-varying'). This choice is currently exposed via the Auto Smooth checkbox.

After a little discussion with @mont29 we figured out that it's probably best to not export vertex normals at all, but to always export loop normals (recomputing on the fly if necessary).

The "Normals" checkbox in the exporter options just enables or disables the exporting of normals. If it's disabled, normals are not written to the ABC file at all. When it's enabled, there are various ways to write the normals to Alembic; currently the ones in use are 'face-varying' and 'vertex-varying'. The thing is, we can only choose one or the other, so we either store loop normals ('face-varying') or vertex normals ('vertex-varying'). This choice is currently exposed via the Auto Smooth checkbox. After a little discussion with @mont29 we figured out that it's probably best to not export vertex normals at all, but to always export loop normals (recomputing on the fly if necessary).

Andrew commented

2019-10-30 16:55:25 +01:00

@dr.sybren Thanks! Makes sense, I think that's a valid approach. Though one downside of saving face varying for every object in the scene -- even for objects that have smooth normals -- is that it could significantly increase the size of the output abc file. Another potentially bigger problem, is that upon re-import, if the tool ingesting that abc is not smart enough to merge identical face-varying normals for a vertex into a vertex-varying normal, that "normals bloat" could make its way to the GPU and really hinder render performance. Imagine a scene filled with thousands of smooth-normaled spheres. Exporting that and reimporting that -- even with normals -- should ideally not result in face-varying normals -- which consume many times the space of vertex-varying ones -- making their way to the GPU memory (or cpu memory if dealing with a cpu renderer).

Some other dcc exporters make one slight tweak to your approach: They examine the normals of each object, and if an object has 100% smooth normals (i.e. they can be trivially recaulculated by an ingest tool via averaging the normals of each vertexs' connected faces), normals are not saved for those objects at all. For those objects that have normals that can only be represented via face-varying, those normals are saved to the abc as face varying. The rationale for this approach is that it keeps the file size as small as possible -- The downside is that the tool ingesting the abc needs to be able to generate smooth normals if necessary (using the connected face averaging approach mentioned above). I think this is actually fine tradeoff. Something to consider.

Useful References on this topic:
https://groups.google.com/d/msg/alembic-discussion/xmP2QMlIWKM/-chYFSPWU_UJ
https://github.com/alembic/alembic/blob/master/maya/AbcExport/MayaMeshWriter.cpp#L659

@dr.sybren Thanks! Makes sense, I think that's a valid approach. Though one downside of saving face varying for every object in the scene -- even for objects that have smooth normals -- is that it could significantly increase the size of the output abc file. Another potentially bigger problem, is that upon re-import, if the tool ingesting that abc is not smart enough to merge identical face-varying normals for a vertex into a vertex-varying normal, that "normals bloat" could make its way to the GPU and really hinder render performance. Imagine a scene filled with thousands of smooth-normaled spheres. Exporting that and reimporting that -- even with normals -- should ideally not result in face-varying normals -- which consume many times the space of vertex-varying ones -- making their way to the GPU memory (or cpu memory if dealing with a cpu renderer). Some other dcc exporters make one slight tweak to your approach: They examine the normals of each object, and if an object has 100% smooth normals (i.e. they can be trivially recaulculated by an ingest tool via averaging the normals of each vertexs' connected faces), normals are not saved for those objects at all. For those objects that have normals that can only be represented via face-varying, those normals are saved to the abc as face varying. The rationale for this approach is that it keeps the file size as small as possible -- The downside is that the tool ingesting the abc needs to be able to generate smooth normals if necessary (using the connected face averaging approach mentioned above). I think this is actually fine tradeoff. Something to consider. Useful References on this topic: https://groups.google.com/d/msg/alembic-discussion/xmP2QMlIWKM/-chYFSPWU_UJ https://github.com/alembic/alembic/blob/master/maya/AbcExport/MayaMeshWriter.cpp#L659

Andrew commented

2019-10-30 16:59:44 +01:00

For such an approach to work and allow blender to have "normals idempotence" with abc files that it outputs and re-imports, blender would need to be (optionally?) capable of generating smooth normals on the fly for objects that don't have normals specified, whenever an abc is imported. Maybe it already has that, but I'm not sure offhand...