FBX export of Face smoothing info (without custom normals) #104434

Open
opened 2023-02-25 10:36:33 +01:00 by Yury · 8 comments

Issue

(Summary from original report and investigations by @Mysteryem, see comments below).

Current export of 'smooth faces' boolean flag, without exporting custom normals, is producing invalid imports in 3DSMax (which seems to interpret them as its 'smooth groups' - non-exclusive bitflags of face groups, each face can belong to more than one smooth groups).

FBX Review does show blender-exported files properly in that case though.

Unity never produces the expected result, regardless of how smooth face info is encoded (as done currently, with the complex bitflag way, etc.).

Original report:

When exporting model as FBX with smoothing option in exporter set to Face, it uses bool attribute of polygon "use_smooth" to define smoothing groups which is pretty useless (or at least I can't imagine a case where it would be useful):

code from export_fbx_bin.py

t_ps = array.array(data_types.ARRAY_INT32, (0,)) * len(me.polygons)
me.polygons.foreach_get("use_smooth", t_ps)

There are ways to calculate smoothing groups from vertex normals when importing to other 3D package, however they came with its flaws (at least 3dsMaxs one)

For example if we have a mesh made in Blender with hard edges set like this:
1.jpg
2.jpg

Export it as FBX with smoothing set to "Normals Only" (because "Face" just gives us one smoothing group without any of normal splits, also i had no luck with "Edge")
After importing in 3dsMax with Smoothing Groups checkbox enabled (in this case 3dsMax can't find smoothing groups inside FBX file and will generate smoothing groups from vertex normals). We get this result.

3.jpg
4.jpg

Notice that 3dsMax created only 2 smoothing groups, which isn't enough to preserve the look of original model (central vertex has no normal split). On more complex models this issue can be more devastating, requiring lots of manual work to fix.

Expected result should be like this:
5.jpg

I modified export_fbx_bin.py so it now can calculate smoothing groups. It works but far from being optimized due to my limited coding skills. I am sure it can be done much better by actual blender-addons devs. Thanks.

export_fbx_bin.py line 973

            # BEGINNING OF SMOOTHING GROUP PATCH

            smoothing_groups = (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32)

            #Calculate face groups (Can't really use_bitflags=True because it can put two face groups into one smoothing group even if they both have common "sharp" vert)
            face_groups = me.calc_smooth_groups(use_bitflags=False)[0]

            # Create dict {vertex_index: {facegroupIDs}}
            v_fg = {}
            for poly in me.polygons:
                for v_index in poly.vertices:        
                    polygroups = v_fg.get(v_index)
                    if not polygroups:
                        polygroups = set()
                    polygroups.add(face_groups[poly.index])
                    v_fg.update({v_index: polygroups})

            #Create dict {facegroupID: [polys]}
            fg_p = {}
            for poly in me.polygons:
                polys = fg_p.get(face_groups[poly.index])
                if not polys:
                    polys = []
                polys.append(poly)
                fg_p.update({face_groups[poly.index]: polys })

            #Create dict {facegroupID: {neighborfacegroupIDs}}
            fg_n = {}
            for fg in fg_p:
                neighbors = set()
                for poly in fg_p[fg]:
                    for v_index in poly.vertices:
                        neighbors.update(v_fg[v_index])
                neighbors.remove(fg)
                fg_n.update({fg: neighbors})
                        
            #Create dict {facegroupID: SmoothingGroupID}
            sg_of_fg = {}
            for fg in fg_n:
                if fg == 1:
                    sg = smoothing_groups[1]
                else:
                    neighbors = fg_n[fg]
                    neighbors_sgs = set()
                    for neighbor in neighbors:           
                        neighbor_sg = sg_of_fg.get(neighbor)
                        neighbors_sgs.add(neighbor_sg)
                    sg = min([sg for sg in smoothing_groups if sg not in neighbors_sgs and sg !=0])             
                sg_of_fg.update({fg: sg})
                
            #Create list [poly_index: SmoothingGroupID]
            smoothing_groups = list(map(sg_of_fg.get, face_groups))
            t_ps = [2**(x-1) for x in smoothing_groups]

            # END OF SMOOTHING GROUP PATCH
            
            # t_ps = array.array(data_types.ARRAY_INT32, (0,)) * len(me.polygons)
            # me.polygons.foreach_get("use_smooth", t_ps)
## Issue *(Summary from original report and investigations by @Mysteryem, see comments below).* Current export of 'smooth faces' boolean flag, without exporting custom normals, is producing invalid imports in 3DSMax (which seems to interpret them as its 'smooth groups' - non-exclusive bitflags of face groups, each face can belong to more than one smooth groups). FBX Review does show blender-exported files properly in that case though. Unity never produces the expected result, regardless of how smooth face info is encoded (as done currently, with the complex bitflag way, etc.). ## Original report: When exporting model as FBX with smoothing option in exporter set to Face, it uses bool attribute of polygon "use_smooth" to define smoothing groups which is pretty useless (or at least I can't imagine a case where it would be useful): code from export_fbx_bin.py ``` t_ps = array.array(data_types.ARRAY_INT32, (0,)) * len(me.polygons) me.polygons.foreach_get("use_smooth", t_ps) ``` There are ways to calculate smoothing groups from vertex normals when importing to other 3D package, however they came with its flaws (at least 3dsMaxs one) For example if we have a mesh made in Blender with hard edges set like this: ![1.jpg](/attachments/8d0a042e-ccf3-4a38-8dfe-4baea83f7845) ![2.jpg](/attachments/18a22c6c-b991-4d53-a6f6-19895c55bbe3) Export it as FBX with smoothing set to "Normals Only" (because "Face" just gives us one smoothing group without any of normal splits, also i had no luck with "Edge") After importing in 3dsMax with Smoothing Groups checkbox enabled (in this case 3dsMax can't find smoothing groups inside FBX file and will generate smoothing groups from vertex normals). We get this result. ![3.jpg](/attachments/2564c314-8363-4bd4-9b96-4ea3efc434b9) ![4.jpg](/attachments/aac3922a-864c-44a2-86a8-0e6371d68370) Notice that 3dsMax created only 2 smoothing groups, which isn't enough to preserve the look of original model (central vertex has no normal split). On more complex models this issue can be more devastating, requiring lots of manual work to fix. Expected result should be like this: ![5.jpg](/attachments/f1dba410-4100-4336-b182-a03d48a7be48) I modified export_fbx_bin.py so it now can calculate smoothing groups. It works but far from being optimized due to my limited coding skills. I am sure it can be done much better by actual blender-addons devs. Thanks. export_fbx_bin.py line 973 ``` # BEGINNING OF SMOOTHING GROUP PATCH smoothing_groups = (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32) #Calculate face groups (Can't really use_bitflags=True because it can put two face groups into one smoothing group even if they both have common "sharp" vert) face_groups = me.calc_smooth_groups(use_bitflags=False)[0] # Create dict {vertex_index: {facegroupIDs}} v_fg = {} for poly in me.polygons: for v_index in poly.vertices: polygroups = v_fg.get(v_index) if not polygroups: polygroups = set() polygroups.add(face_groups[poly.index]) v_fg.update({v_index: polygroups}) #Create dict {facegroupID: [polys]} fg_p = {} for poly in me.polygons: polys = fg_p.get(face_groups[poly.index]) if not polys: polys = [] polys.append(poly) fg_p.update({face_groups[poly.index]: polys }) #Create dict {facegroupID: {neighborfacegroupIDs}} fg_n = {} for fg in fg_p: neighbors = set() for poly in fg_p[fg]: for v_index in poly.vertices: neighbors.update(v_fg[v_index]) neighbors.remove(fg) fg_n.update({fg: neighbors}) #Create dict {facegroupID: SmoothingGroupID} sg_of_fg = {} for fg in fg_n: if fg == 1: sg = smoothing_groups[1] else: neighbors = fg_n[fg] neighbors_sgs = set() for neighbor in neighbors: neighbor_sg = sg_of_fg.get(neighbor) neighbors_sgs.add(neighbor_sg) sg = min([sg for sg in smoothing_groups if sg not in neighbors_sgs and sg !=0]) sg_of_fg.update({fg: sg}) #Create list [poly_index: SmoothingGroupID] smoothing_groups = list(map(sg_of_fg.get, face_groups)) t_ps = [2**(x-1) for x in smoothing_groups] # END OF SMOOTHING GROUP PATCH # t_ps = array.array(data_types.ARRAY_INT32, (0,)) * len(me.polygons) # me.polygons.foreach_get("use_smooth", t_ps) ```
46 KiB
62 KiB
37 KiB
38 KiB
26 KiB
Contributor

@mont29 can you please take a look?

@mont29 can you please take a look?

@Andrej-4 not sure whether the smoothing groups of FBX are supposed/expected to follow the smoothing groups handling of 3DSMax? Would be good to get evidence of this first.

Besides that, code here was very recently (speed-)improved by using numpy, so maybe @Mysteryem would be interested in this?

@Andrej-4 not sure whether the smoothing groups of FBX are supposed/expected to follow the smoothing groups handling of 3DSMax? Would be good to get evidence of this first. Besides that, code here was very recently (speed-)improved by using numpy, so maybe @Mysteryem would be interested in this?
Bastien Montagne added the
Type
To Do
Interest
Pipeline, Assets & IO
Status
Confirmed
labels 2023-03-06 16:20:13 +01:00
Member

I don't have 3DSMAX, but I had a try with Unity and FBX Review.
I modified the exporter to disabled the export of normals, so that normals would have to be calculated from smoothing groups, and then imported FBX exported using the above code (left) as well as t_ps = me.calc_smooth_groups(use_bitflags=True)[0] (middle) and t_ps = me.calc_smooth_groups(use_bitflags=False)[0] (right) into Unity:
image
I can only guess that Unity is interpreting the data as something else or its support is broken. I tried 2019, 2020 and the 2023 alpha.

I also opened all three in FBX Review, but each .fbx would be completely smooth.
image

Blender's current export of the "use_smooth" attribute however does work as expected in FBX Review (still with normals excluded from the export). Unity on the other hand is still doing who knows what:
image
It looks like FBX Review is treating the smoothing like a bool in the same way as the FBX exporter is currently, which would explain why the previous 3 .fbx all were entirely smooth since the smoothing groups start at 1.

I don't have 3DSMAX, but I had a try with Unity and FBX Review. I modified the exporter to disabled the export of normals, so that normals would have to be calculated from smoothing groups, and then imported FBX exported using the above code (left) as well as `t_ps = me.calc_smooth_groups(use_bitflags=True)[0]` (middle) and `t_ps = me.calc_smooth_groups(use_bitflags=False)[0]` (right) into Unity: ![image](/attachments/55f17c30-f3c7-4176-b603-d51f4f7bf141) I can only guess that Unity is interpreting the data as something else or its support is broken. I tried 2019, 2020 and the 2023 alpha. I also opened all three in FBX Review, but each .fbx would be completely smooth. ![image](/attachments/500b4113-e33b-4b62-9e12-c4252ba97287) Blender's current export of the "use_smooth" attribute however does work as expected in FBX Review (still with normals excluded from the export). Unity on the other hand is still doing who knows what: ![image](/attachments/6023dcb8-ab6d-4bf1-b5fc-d7ecf9084c94) It looks like FBX Review is treating the smoothing like a bool in the same way as the FBX exporter is currently, which would explain why the previous 3 .fbx all were entirely smooth since the smoothing groups start at 1.

Thanks for the investigation @Mysteryem!

Am afraid that in the current state of FBX, only conclusion we can draw here is that the reliable way to get this info out is to use export normals. :(

Thanks for the investigation @Mysteryem! Am afraid that in the current state of FBX, only conclusion we can draw here is that the reliable way to get this info out is to use export normals. :(
Bastien Montagne added
Type
Known Issue
and removed
Type
To Do
labels 2023-03-07 09:55:56 +01:00
Bastien Montagne changed title from FBX Face smoothing not exporting proper smoothing groups to FBX export of Face smoothing info (without custom normals) 2023-03-07 09:56:54 +01:00
Author

@Mysteryem if you assign me.calc_smooth_groups(use_bitflags=True)[0] directly to t_ps it will produce unwanted results because of how FBX is storing smoothing groups. For example group 1 will be stored in fbx as 1, 2 as 2, 3 as 4, 4 as 8, 5 as 16,... 13 as 4096 etc.
Something like this should work for your testing needs
smoothing_groups = me.calc_smooth_groups(use_bitflags=True)[0] t_ps = [2**(x-1) for x in smoothing_groups]

@Mysteryem if you assign `me.calc_smooth_groups(use_bitflags=True)[0]` directly to `t_ps` it will produce unwanted results because of how FBX is storing smoothing groups. For example group 1 will be stored in fbx as 1, 2 as 2, 3 as 4, 4 as 8, 5 as 16,... 13 as 4096 etc. Something like this should work for your testing needs `smoothing_groups = me.calc_smooth_groups(use_bitflags=True)[0] t_ps = [2**(x-1) for x in smoothing_groups]`
Member

@Mysteryem if you assign me.calc_smooth_groups(use_bitflags=True)[0] directly to t_ps it will produce unwanted results because of how FBX is storing smoothing groups. For example group 1 will be stored in fbx as 1, 2 as 2, 3 as 4, 4 as 8, 5 as 16,... 13 as 4096 etc.
Something like this should work for your testing needs
smoothing_groups = me.calc_smooth_groups(use_bitflags=True)[0] t_ps = [2**(x-1) for x in smoothing_groups]

Both me.calc_smooth_groups(use_bitflags=True)[0] and the code in the issue produce values that are powers of 2, e.g. [1, 2, 4, 8, 16, ..., 4096]. That makes sense to me, the FBX values are 32 bit integers and there are 32 available smoothing groups, giving a bitmask of 32 bits. So assigning a face to both smoothing group 1 and 2 could be done with 1 | 2 => 3.

This should also mean that if I manually set every smoothing value to -1 (Blender's FBX exporter code uses signed integers internally), it would set every polygon into all smoothing groups. However, when I open such a file in FBX Review it is displayed entirely flat shaded, as if the value had been clamped to zero or higher and then read as a bool. Trying with other negative numbers produces the same results.

Unity however imports such .fbx as entirely smooth as would be expected, since all the negative numbers will have at least one bit in the same position that is set to 1.

I get the feeling that Unity is interpreting the values as bitmasks of smoothing groups as expected, but is applying them to the wrong parts of the mesh.

I did manage to find an old FBX 2013.1 ascii fbx with smoothing groups. I deleted the normals, but FBX Review would still only treat the smoothing groups as bools. Perhaps there's something else that has to be set in FBX for the values to be treated as bitmasks rather than bools or perhaps smoothing groups were deprecated at some point, I really don't know.

> @Mysteryem if you assign `me.calc_smooth_groups(use_bitflags=True)[0]` directly to `t_ps` it will produce unwanted results because of how FBX is storing smoothing groups. For example group 1 will be stored in fbx as 1, 2 as 2, 3 as 4, 4 as 8, 5 as 16,... 13 as 4096 etc. > Something like this should work for your testing needs > `smoothing_groups = me.calc_smooth_groups(use_bitflags=True)[0] > t_ps = [2**(x-1) for x in smoothing_groups]` Both `me.calc_smooth_groups(use_bitflags=True)[0]` and the code in the issue produce values that are powers of 2, e.g. `[1, 2, 4, 8, 16, ..., 4096]`. That makes sense to me, the FBX values are 32 bit integers and there are 32 available smoothing groups, giving a bitmask of 32 bits. So assigning a face to both smoothing group 1 and 2 could be done with `1 | 2 => 3`. This should also mean that if I manually set every smoothing value to -1 (Blender's FBX exporter code uses signed integers internally), it would set every polygon into all smoothing groups. However, when I open such a file in FBX Review it is displayed entirely flat shaded, as if the value had been clamped to zero or higher and then read as a bool. Trying with other negative numbers produces the same results. Unity however imports such .fbx as entirely smooth as would be expected, since all the negative numbers will have at least one bit in the same position that is set to 1. I get the feeling that Unity is interpreting the values as bitmasks of smoothing groups as expected, but is applying them to the wrong parts of the mesh. I did manage to find an old FBX 2013.1 ascii fbx with smoothing groups. I deleted the normals, but FBX Review would still only treat the smoothing groups as bools. Perhaps there's something else that has to be set in FBX for the values to be treated as bitmasks rather than bools or perhaps smoothing groups were deprecated at some point, I really don't know.
Author

@Andrej-4 not sure whether the smoothing groups of FBX are supposed/expected to follow the smoothing groups handling of 3DSMax? Would be good to get evidence of this first.

Besides that, code here was very recently (speed-)improved by using numpy, so maybe @Mysteryem would be interested in this?

I just tested it with Unreal Engine with "Normal Input Method" set as "Compute Normals" which computes normals from smoothing groups. I had 2 identical meshes one of which was exported with smoothing groups (through edited export_fbx_bin.py) and other with "Normals Only". FBX with smoothing groups imported perfectly in UE while FBX with normals caused an error in UE:

No smoothing group information was found in this FBX scene. Please make sure to enable the 'Export Smoothing Groups' option in the FBX Exporter plug-in before exporting the file. Even for tools that don't support smoothing groups, the FBX Exporter will generate appropriate smoothing data at export-time so that correct vertex normals can be inferred while importing.

UE assumes that we have Smoothing Group export in our 3d package which is sadly not true.

Answering @mont29 question, I am pretty sure i got FBX smoothing groups right, i use edited exporter daily in clients work (i do 3d for a living) and everything seems to be working just fine.

Also as my scripting skills grow(hopefully) i enhanced the edits to the code a little and now it's using numpy for the t_ps write!

export_fbx_bin.py Line 1031

        if smooth_type == 'FACE':
            # BEGINNING OF SMOOTHING GROUP PATCH
            from collections import defaultdict
            smoothing_groups = tuple(range(33))
            face_groups = me.calc_smooth_groups(use_bitflags=False)[0]
            # Create dict {vertex_index: {facegroupIDs}}    
            v_fg = defaultdict(set)
            for poly in me.polygons:
                for v_index in poly.vertices:
                    v_fg[v_index].add(face_groups[poly.index])
            # Create dict {facegroupID: [polys]}
            fg_p = defaultdict(list)
            for poly in me.polygons:
                fg_p[face_groups[poly.index]].append(poly)
            # Create dict {facegroupID: {neighborfacegroupIDs}}
            fg_n = {fg: {n for poly in fg_p[fg] for v_index in poly.vertices for n in v_fg[v_index]} - {fg} for fg in fg_p}
            # Create dict {facegroupID: SmoothingGroupID}
            sg_of_fg = {}
            for fg in fg_n:
                if fg == 1:
                    sg = smoothing_groups[1]
                else:
                    neighbors = fg_n[fg]
                    neighbors_sgs = set()
                    for neighbor in neighbors:
                        neighbor_sg = sg_of_fg.get(neighbor)
                        neighbors_sgs.add(neighbor_sg)
                    sg = min([sg for sg in smoothing_groups if sg not in neighbors_sgs and sg != 0])
                sg_of_fg.update({fg: sg})
            t_ps = np.array([2**(sg_of_fg.get(fg) - 1) for fg in face_groups])
            _map = b"ByPolygon"
        # END OF SMOOTHING GROUP PATCH
            # t_ps = np.empty(len(me.polygons), dtype=poly_use_smooth_dtype)
            # me.polygons.foreach_get("use_smooth", t_ps)
            # _map = b"ByPolygon"
        else:  # EDGE

As i previously said i use modified exporter on a daily basis, tested with 500k tris meshes, time added for the sg calculation is barely noticeable and I am sure it is still can be improved by a lot.

I really hope someone will look into this and maybe add it to the FBX exporter because smoothing groups are often (at my work ALWAYS) expected in 3d meshes and it will for sure increase Blender's value as a tool in complex pipelines.

> @Andrej-4 not sure whether the smoothing groups of FBX are supposed/expected to follow the smoothing groups handling of 3DSMax? Would be good to get evidence of this first. > > Besides that, code here was very recently (speed-)improved by using numpy, so maybe @Mysteryem would be interested in this? I just tested it with Unreal Engine with "Normal Input Method" set as "Compute Normals" which computes normals from smoothing groups. I had 2 identical meshes one of which was exported with smoothing groups (through edited export_fbx_bin.py) and other with "Normals Only". FBX with smoothing groups imported perfectly in UE while FBX with normals caused an error in UE: `No smoothing group information was found in this FBX scene. Please make sure to enable the 'Export Smoothing Groups' option in the FBX Exporter plug-in before exporting the file. Even for tools that don't support smoothing groups, the FBX Exporter will generate appropriate smoothing data at export-time so that correct vertex normals can be inferred while importing. ` UE assumes that we have Smoothing Group export in our 3d package which is sadly not true. Answering @mont29 question, I am pretty sure i got FBX smoothing groups right, i use edited exporter daily in clients work (i do 3d for a living) and everything seems to be working just fine. Also as my scripting skills grow(hopefully) i enhanced the edits to the code a little and now it's using numpy for the t_ps write! export_fbx_bin.py Line 1031 ``` if smooth_type == 'FACE': # BEGINNING OF SMOOTHING GROUP PATCH from collections import defaultdict smoothing_groups = tuple(range(33)) face_groups = me.calc_smooth_groups(use_bitflags=False)[0] # Create dict {vertex_index: {facegroupIDs}} v_fg = defaultdict(set) for poly in me.polygons: for v_index in poly.vertices: v_fg[v_index].add(face_groups[poly.index]) # Create dict {facegroupID: [polys]} fg_p = defaultdict(list) for poly in me.polygons: fg_p[face_groups[poly.index]].append(poly) # Create dict {facegroupID: {neighborfacegroupIDs}} fg_n = {fg: {n for poly in fg_p[fg] for v_index in poly.vertices for n in v_fg[v_index]} - {fg} for fg in fg_p} # Create dict {facegroupID: SmoothingGroupID} sg_of_fg = {} for fg in fg_n: if fg == 1: sg = smoothing_groups[1] else: neighbors = fg_n[fg] neighbors_sgs = set() for neighbor in neighbors: neighbor_sg = sg_of_fg.get(neighbor) neighbors_sgs.add(neighbor_sg) sg = min([sg for sg in smoothing_groups if sg not in neighbors_sgs and sg != 0]) sg_of_fg.update({fg: sg}) t_ps = np.array([2**(sg_of_fg.get(fg) - 1) for fg in face_groups]) _map = b"ByPolygon" # END OF SMOOTHING GROUP PATCH # t_ps = np.empty(len(me.polygons), dtype=poly_use_smooth_dtype) # me.polygons.foreach_get("use_smooth", t_ps) # _map = b"ByPolygon" else: # EDGE ``` As i previously said i use modified exporter on a daily basis, tested with 500k tris meshes, time added for the sg calculation is barely noticeable and I am sure it is still can be improved by a lot. I really hope someone will look into this and maybe add it to the FBX exporter because smoothing groups are often (at my work ALWAYS) expected in 3d meshes and it will for sure increase Blender's value as a tool in complex pipelines.
Author

I would greatly appreciate any updates on this matter, as with every new Blender release, I find myself needing to modify the FBX exporter for both my team and myself.

While I can't speak for the entire game development community, in the company I work for (which consists of over 300 artists, with roughly 20% using Blender), Blender artists often struggle with this step. The majority of models still need to be saved in 3ds Max with smoothing groups.

I believe having an option to export correct smoothing groups is crucial.If you need any information/proofs about how it works in 3ds Max or in FBX in general, just let me know. I'll do my best to provide the details you need.

I would greatly appreciate any updates on this matter, as with every new Blender release, I find myself needing to modify the FBX exporter for both my team and myself. While I can't speak for the entire game development community, in the company I work for (which consists of over 300 artists, with roughly 20% using Blender), Blender artists often struggle with this step. The majority of models still need to be saved in 3ds Max with smoothing groups. I believe having an option to export correct smoothing groups is crucial.If you need any information/proofs about how it works in 3ds Max or in FBX in general, just let me know. I'll do my best to provide the details you need.
Sign in to join this conversation.
No Milestone
No project
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender-addons#104434
No description provided.