Obj import generate some not valid meshes #119823

New Issue

Open

opened 2024-03-23 14:01:37 +01:00 by Julien Duroure · 16 comments

Julien Duroure commented

2024-03-23 14:01:37 +01:00

Member

System Information
Operating system: Mint
Graphics card: Nvidia

Blender Version
Broken: current build 4.2 alpha
Broken: current build 4.1 candidate
Broken: current 3.6.10 LTS
Worked: ?

Short description of error
Obj import generate some not valid meshes.
With this file : https://casual-effects.com/g3d/data10/research/model/San_Miguel/San_Miguel.zip.

Exact steps for others to reproduce the error

Import obj file.
Check it using data.validate() API.

import bpy

for obj in bpy.data.objects:
    if obj.data.validate() is True:
        print("{} object has invalid mesh".format(obj.name))

This was first reported in glTF repository (here https://github.com/KhronosGroup/glTF-Blender-IO/issues/2171), but the root issue is an invalid mesh created by obj importer.

**System Information** Operating system: Mint Graphics card: Nvidia **Blender Version** Broken: current build 4.2 alpha Broken: current build 4.1 candidate Broken: current 3.6.10 LTS Worked: ? **Short description of error** Obj import generate some not valid meshes. With this file : https://casual-effects.com/g3d/data10/research/model/San_Miguel/San_Miguel.zip. **Exact steps for others to reproduce the error** 1. Import obj file. 2. Check it using data.validate() API. ```Py import bpy for obj in bpy.data.objects: if obj.data.validate() is True: print("{} object has invalid mesh".format(obj.name)) ``` This was first reported in glTF repository (here https://github.com/KhronosGroup/glTF-Blender-IO/issues/2171), but the root issue is an invalid mesh created by obj importer.

Julien Duroure added the

labels 2024-03-23 14:01:48 +01:00

Philipp Oeser commented

2024-03-25 09:51:26 +01:00

Member

Can confirm, will check in more detail...

Can confirm, will check in more detail...

Philipp Oeser added

Pipeline, Assets & IO

and removed

labels 2024-03-25 09:51:51 +01:00

Philipp Oeser commented

2024-03-25 11:06:45 +01:00

Member

Will still try to strip down the OBJ to a minimal example (and provide info where it fails), but will already raise the priority so this does not slip under the carpet

Will still try to strip down the OBJ to a minimal example (and provide info where it fails), but will already raise the priority so this does not slip under the carpet

Philipp Oeser added

and removed

labels 2024-03-25 11:06:55 +01:00

Philipp Oeser commented

2024-03-25 11:27:59 +01:00

Member

Can be reproduced with just the Portraits object from the original OBJ

Noting that we of course have the Validate Meshes option which gets rid of the corruptions (still think this deserves to be looked at since that option is OFF by default)

Can be reproduced with just the `Portraits` object from the original OBJ Noting that we of course have the Validate Meshes option which gets rid of the corruptions (still think this deserves to be looked at since that option is OFF by default)

san-miguel-low-poly-portraits.obj

164 KiB

Philipp Oeser commented

2024-03-25 11:29:37 +01:00

Member

CC @aras_p

Aras Pranckevicius self-assigned this 2024-03-25 11:41:32 +01:00

Aras Pranckevicius commented

2024-03-25 11:59:38 +01:00

Member

The issue that the OBJ in question does have literally duplicate faces, which in Blender is considered "invalid". E.g. from Portraits repro that Philipp made (around line 3925):

f 121/20/1 122/2/1 123/3/1 ## this one
f 124/21/1 122/2/1 121/20/1
f 125/20/2 126/3/2 127/2/2
f 128/21/2 125/20/2 127/2/2
f 121/20/1 122/2/1 123/3/1 ## and this one, literally same verts

Now, originally I made "validate" option be off since it is very costly, and I went with how already existing other importers did (Alembic has option to validate, which is off by default; whereas USD never validates meshes).

The issue that the OBJ in question *does* have literally duplicate faces, which in Blender is considered "invalid". E.g. from `Portraits` repro that Philipp made (around line 3925): ``` f 121/20/1 122/2/1 123/3/1 ## this one f 124/21/1 122/2/1 121/20/1 f 125/20/2 126/3/2 127/2/2 f 128/21/2 125/20/2 127/2/2 f 121/20/1 122/2/1 123/3/1 ## and this one, literally same verts ``` Now, originally I made "validate" option be off since it is *very* costly, and I went with how already existing other importers did (Alembic has option to validate, which is off by default; whereas USD *never* validates meshes).

Philipp Oeser commented

2024-03-25 12:19:07 +01:00

Member

OK, thx checking @aras_p

I also asked in #pipeline-assets-io-module what the best approach here would be:

enable Validate Meshes by default (costly)?
spend time making the importers more robust (could also end up being costly)?
leave as is?

I would also vote for making Validate Meshes a proper operator that could be accesses via a menu entry, I use the API almost on a daily basis

OK, thx checking @aras_p I also asked in #pipeline-assets-io-module what the best approach here would be: - enable `Validate Meshes` by default (costly)? - spend time making the importers more robust (could also end up being costly)? - leave as is? I would also vote for making `Validate Meshes` a proper operator that could be accesses via a menu entry, I use the API almost on a daily basis

Scurest commented

2024-03-25 15:10:02 +01:00

Contributor

Note that the glTF exporter already calls validate on every object it exports.

e21b4d4bfd/io_scene_gltf2/blender/exp/gltf2_blender_gather_nodes.py (L251)

This shows that, at least in this workflow, the cost of validation isn't gone, just moved around, and is in fact paid double if the mesh was already validated. But more pertinently, it means just calling validate on this mesh does not make the glTF export work. OTOH if you use the "Validate Meshes" option when importing, the export will work. Something else is going on.

Import san-miguel-low-poly-portraits.obj from the UI, then run this script

import bpy
 
m = bpy.context.active_object.data

print(len(m.loops), len(m.loop_triangles), 3*len(m.loop_triangles))
print(m.validate())
print(len(m.loops), len(m.loop_triangles), 3*len(m.loop_triangles))

arr = [0] * (3 * len(m.loop_triangles))
try:
    m.loop_triangles.foreach_get('loops', arr)
except Exception as e:
    print('Error:', e)

Importing with "Validate Meshes" off (default) produces

5304 1768 5304
True
5016 1672 5016
Error: Array length mismatch (expected 5304, got 5016)
Error: internal error setting the array

while importing with "Validate Meshes" on produces

5016 1672 5016
False
5016 1672 5016

So my guess is the real root cause of the export not working is that validate() does not invalidate loop_triangles when it changes the mesh.

To test this, I tried importing and then exporting in the same script. It works, presumably because you don't see the mesh in the viewport, which would evaluate loop_triangles, so loop_triangles is not calculated until the glTF exporter accesses it, which is after it calls validate. If you do something to force loop_triangles to be evaluated in between the import and export (ie before the validate call), the export stops working again.

import bpy

bpy.ops.wm.obj_import(filepath='san-miguel-low-poly-portraits.obj')

# If this is uncommented, the export doesn't work
#for tri in bpy.context.active_object.data.loop_triangles: print(tri.loops)

bpy.ops.export_scene.gltf(filepath='out.glb')

Note that the glTF exporter already calls validate on every object it exports. https://projects.blender.org/blender/blender-addons/src/commit/e21b4d4bfdc74726a0848ad7eb61ecc1eaa1176a/io_scene_gltf2/blender/exp/gltf2_blender_gather_nodes.py#L251 This shows that, at least in this workflow, the cost of validation isn't gone, just moved around, and is in fact paid double if the mesh was already validated. But more pertinently, it means just calling validate on this mesh does not make the glTF export work. OTOH if you use the "Validate Meshes" option when importing, the export _will_ work. Something else is going on. Import san-miguel-low-poly-portraits.obj from the UI, then run this script ```py import bpy m = bpy.context.active_object.data print(len(m.loops), len(m.loop_triangles), 3*len(m.loop_triangles)) print(m.validate()) print(len(m.loops), len(m.loop_triangles), 3*len(m.loop_triangles)) arr = [0] * (3 * len(m.loop_triangles)) try: m.loop_triangles.foreach_get('loops', arr) except Exception as e: print('Error:', e) ``` Importing with "Validate Meshes" off (default) produces ``` 5304 1768 5304 True 5016 1672 5016 Error: Array length mismatch (expected 5304, got 5016) Error: internal error setting the array ``` while importing with "Validate Meshes" on produces ``` 5016 1672 5016 False 5016 1672 5016 ``` So my guess is the _real_ root cause of the export not working is that **`validate()` does not invalidate loop_triangles when it changes the mesh**. To test this, I tried importing and then exporting in the same script. It works, presumably because you don't see the mesh in the viewport, which would evaluate loop_triangles, so loop_triangles is not calculated until the glTF exporter accesses it, which is after it calls validate. If you do something to force loop_triangles to be evaluated in between the import and export (ie before the validate call), the export stops working again. ```py import bpy bpy.ops.wm.obj_import(filepath='san-miguel-low-poly-portraits.obj') # If this is uncommented, the export doesn't work #for tri in bpy.context.active_object.data.loop_triangles: print(tri.loops) bpy.ops.export_scene.gltf(filepath='out.glb') ```

Bastien Montagne added

and removed

labels 2024-03-29 02:00:44 +01:00

Bastien Montagne commented

2024-03-29 02:01:11 +01:00

Owner

This is not a new issue, and not a regression, no reason for this to be high prio.

This is not a new issue, and not a regression, no reason for this to be high prio.

Scurest commented

2024-03-29 05:50:53 +01:00

Contributor

The fact that the OBJ import -> glTF export does not work is a regression. It worked in 3.6.

The fact that the OBJ import -> glTF export does not work is a regression. It worked in 3.6.

Bastien Montagne commented

2024-03-29 12:49:11 +01:00

Owner

But was the mesh generated by OBJ import valid in 3.6?

Saying that it's a regression does not really help by itself, we need to understand what is actually happening. Given current info, the OBJ data itself is invalid. So the fact that at some point handling this invalid data worked for some reason does not mean that it not working anymore is a regression by itself.

But was the mesh generated by OBJ import valid in 3.6? Saying that it's a regression does not really help by itself, we need to understand what is actually happening. Given current info, the OBJ data itself is invalid. So the fact that at some point handling this invalid data worked for some reason does not mean that it not working anymore is a regression by itself.

Scurest commented

2024-03-29 15:18:52 +01:00

Contributor

The OBJ import was not valid in 3.6.

Again, the fact that the import->export worked in 3.6 despite the import also being invalid, and the fact that the glTF exporter itself calls validate on every mesh it exports, and the fact that the export works when the OBJ importer calls validate but not when the glTF exporter does, means that the mesh being invalid is not itself the problem here.

The problem is that mesh.validate() does not appear to purge cached loop_triangles data, so accessing loop_triangles after calling validate gives the stale data from when the mesh was invalid.

If you do something to force loop_triangles to be recalculated after a validate call, like toggle in and out of edit mode, or save and reopen the .blend file, it will work again.

The OBJ import was not valid in 3.6. Again, the fact that the import->export worked in 3.6 despite the import also being invalid, and the fact that the glTF exporter itself calls validate on every mesh it exports, and the fact that the export works when the OBJ importer calls validate but not when the glTF exporter does, means that the mesh being invalid is not itself the problem here. The problem is that `mesh.validate()` does not appear to purge cached loop_triangles data, so accessing loop_triangles after calling validate gives the stale data from when the mesh was invalid. If you do something to force loop_triangles to be recalculated after a validate call, like toggle in and out of edit mode, or save and reopen the .blend file, it will work again.

Aras Pranckevicius commented

2024-03-29 15:36:31 +01:00

Member

@scurest I have not validated (ha :)) this, but if you say so, then to me it sounds like neither OBJ nor glTF are directly related, but actual issue/regression would be along the lines of "in blender 3.6, mesh.validate() actually fixed it, but now it keeps garbage cached loop_triangles data and so the mesh stays invalid after validate call". If my understanding of this is correct, then I'd defer to @HooglyBoogly perhaps, maybe all the mesh data changes after 3.6 broke something, or similar.

@scurest I have not validated (ha :)) this, but if you say so, then to me it sounds like neither OBJ nor glTF are directly related, but actual issue/regression would be along the lines of "in blender 3.6, mesh.validate() actually fixed it, but now it keeps garbage cached loop_triangles data and so the mesh stays invalid after validate call". If my understanding of this is correct, then I'd defer to @HooglyBoogly perhaps, maybe all the mesh data changes after 3.6 broke something, or similar.

Scurest commented

2024-03-29 15:46:50 +01:00

Contributor

@aras_p Yes, you can demonstrate this without either glTF or OBJ. For example, open this file (from this Q) and call validate on the single invalid mesh in it (via the Python console). The visual appearance in the 3D view breaks. If you do something to force loop_triangles to be recalculated, it's fixed. Calling validate before it is rendered in the 3D view (thus presumably before loop_triangles is calculated), or trying this in 3.6, also does not manifest the issue.

The change since 3.6 is probably related to the fact loop_triangles is no longer explicitly calculated via calc_loop_triangles() but is calculated on demand.

@aras_p Yes, you can demonstrate this without either glTF or OBJ. For example, open [this file](https://blend-exchange.com/b/BgSVpVdy/) (from [this Q](https://blender.stackexchange.com/questions/315002/3d-object-from-blender-to-gltf-not-showing-why)) and call validate on the single invalid mesh in it (via the Python console). The visual appearance in the 3D view breaks. If you do something to force loop_triangles to be recalculated, it's fixed. Calling validate before it is rendered in the 3D view (thus presumably before loop_triangles is calculated), or trying this in 3.6, also does not manifest the issue. ![](https://i.stack.imgur.com/NFgBi.jpg) The change since 3.6 is probably related to the fact loop_triangles is no longer explicitly calculated via `calc_loop_triangles()` but is calculated on demand.

Scurest referenced this issue

2024-04-02 02:20:41 +02:00

Mesh has invalid loop_triangles after validation #120152

Aras Pranckevicius commented

2024-04-03 07:09:07 +02:00

Member

So this should be fixed by #120159, right?

So this should be fixed by #120159, right?

Scurest commented

2024-04-05 05:33:18 +02:00

Contributor

Can confirm that with latest nightly, import san-miguel-low-poly-portraits.obj -> export glTF works again. So #120159 has fixed the original issue.

Note that this was actually noticed in 4.0, but the glTF exporter silently swallows the error. That try/except should be removed so it doesn't hide bugs in Blender.

Can confirm that with latest nightly, import san-miguel-low-poly-portraits.obj -> export glTF works again. So #120159 has fixed the original issue. Note that this was actually noticed in 4.0, but the glTF exporter [silently swallows the error](https://github.com/KhronosGroup/glTF-Blender-IO/blob/a6c00e638f2c646b899504c8e4c0bd89161fc6bc/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_primitives_extract.py#L364-L373). That try/except should be removed so it doesn't hide bugs in Blender.

Jonathan Chemla commented

2024-04-12 12:26:09 +02:00

I would also vote for making Validate Meshes a proper operator that could be accesses via a menu entry, I use the API almost on a daily basis

Is there another thread to track if this validate mesh data would be implemented as an operator? Might indeed be useful, thanks!

The problem is that mesh.validate() does not appear to purge cached loop_triangles data, so accessing loop_triangles after calling validate gives the stale data from when the mesh was invalid.
If you do something to force loop_triangles to be recalculated after a validate call, like toggle in and out of edit mode, or save and reopen the .blend file, it will work again.

Can confirm this behavior on a fresh 4.1.0 install with one of our non-valid mesh which has both unused loops and faces using same vertices. When doing in console mesh.validate(), I then need to toggle/detoggle Edit Mode so that gltf-export successfully export without crash. Note that other exporters like fbx-export seem to take care of that themselves.

> I would also vote for making `Validate Meshes` a proper operator that could be accesses via a menu entry, I use the API almost on a daily basis Is there another thread to track if this `validate mesh data` would be implemented as an operator? Might indeed be useful, thanks! > The problem is that mesh.validate() does not appear to purge cached loop_triangles data, so accessing loop_triangles after calling validate gives the stale data from when the mesh was invalid. > If you do something to force loop_triangles to be recalculated after a validate call, like toggle in and out of edit mode, or save and reopen the .blend file, it will work again. Can confirm this behavior on a fresh 4.1.0 install with one of our non-valid mesh which has both unused loops and faces using same vertices. When doing in console `mesh.validate()`, I then need to toggle/detoggle Edit Mode so that gltf-export successfully export without crash. Note that other exporters like fbx-export seem to take care of that themselves.

Sign in to join this conversation.

No Label

Animation & Rigging

Asset Browser Project Overview

Automated Testing

Blender Asset Bundle

Dependency Graph

Development Management

EEVEE & Viewport

Images & Movies

Motion Tracking

Nodes & Physics

Pipeline, Assets & IO

Platforms, Builds & Tests

Render & Cycles

Render Pipeline

Sculpt, Paint & Texture

Video Sequencer

Virtual Reality

Blender 2.8 Project

Milestone 1: Basic, Local Asset Browser

Good First Issue

Animation & Rigging

Development Management

EEVEE & Viewport

Nodes & Physics

Pipeline, Assets & IO

Platforms, Builds & Tests

Render & Cycles

Sculpt, Paint & Texture

Needs Info from Developers

Needs Information from User

No Milestone

No project

No Assignees

6 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#119823

No description provided.