Mesh import/export with high vertex count is very slow #53236

Closed
opened 2017-11-02 17:57:19 +01:00 by Adam · 28 comments

System Information
Win 7 AMD HD 78xx

Blender Version
Broken: always

Short description of error
Exporting or importing meshes from Blender which have a high number of vertives (> 1 million) take ages to import or export.

Exact steps for others to reproduce the error

  • Create Suzanne
  • Use Subsurf until the polycount is in the millions ( 5 -10 million)
  • Export the mesh (obj or fbx - without UVs/normals will be faster, naturally)
  • Wait forever ;)

Importing the mesh will take an equally long time.

This is a giant blocker to sculpting workflow in Blender as it makes exporting to for example xNormal extremely time consuming.

**System Information** Win 7 AMD HD 78xx **Blender Version** Broken: always **Short description of error** Exporting or importing meshes from Blender which have a high number of vertives (> 1 million) take ages to import or export. **Exact steps for others to reproduce the error** - Create Suzanne - Use Subsurf until the polycount is in the millions ( 5 -10 million) - Export the mesh (obj or fbx - without UVs/normals will be faster, naturally) - Wait forever ;) Importing the mesh will take an equally long time. This is a giant blocker to sculpting workflow in Blender as it makes exporting to for example xNormal extremely time consuming.
Author

Changed status to: 'Open'

Changed status to: 'Open'
Author

Added subscriber: @Win

Added subscriber: @Win
Author
[T53236_MeshExport.blend](https://archive.blender.org/developer/F1087343/T53236_MeshExport.blend)

Added subscriber: @PeterBoos

Added subscriber: @PeterBoos

Changed status from 'Open' to: 'Archived'

Changed status from 'Open' to: 'Archived'
Peter Boos self-assigned this 2017-11-02 19:18:38 +01:00

The art of sculpting isnt super small polygons... Its knowing where to put details and where to remove them.
So if your bad ad sculpting then that comes with a cost.
Its not Blender's fault that processing lots of faces take more time, its mostly the speed of your hardware, you could buy SSD's and a faster computer, or better learn to sculpt.

Like for example suzane use subdiv 2 and use smooth surfaces, a final render would look the same.

The art of sculpting isnt super small polygons... Its knowing where to put details and where to remove them. So if your bad ad sculpting then that comes with a cost. Its not Blender's fault that processing lots of faces take more time, its mostly the speed of your hardware, you could buy SSD's and a faster computer, or better learn to sculpt. Like for example suzane use subdiv 2 and use smooth surfaces, a final render would look the same.
Author

I don't understand your charge that I am bad at sculpting. For highly detailed sculpts resolutions of well over 10 million points are common.

That this is not Blender's fault is also wrong. Other software (take zBrush for example) handles large mesh exports in reasonable timeframes of under 30 seconds, not up to 15 minutes.

And of course I'm aware that making a 10 mil vert suzanne is pointless. It was just an example.

Sculpting is also not the only area that the long load times pose an issue. It happens with photogrammetry generated meshes as well.

Please reopen the task, this should be adressed.

I don't understand your charge that I am bad at sculpting. For highly detailed sculpts resolutions of well over 10 million points are common. That this is not Blender's fault is also wrong. Other software (take zBrush for example) handles large mesh exports in reasonable timeframes of under 30 seconds, not up to 15 minutes. And of course I'm aware that making a 10 mil vert suzanne is pointless. It was just an example. Sculpting is also not the only area that the long load times pose an issue. It happens with photogrammetry generated meshes as well. Please reopen the task, this should be adressed.

If this is your regular job, then i stronlgy recommend you a SSD, this is not a bug in code; the waiting time you have is simply caused by the IO speed of your harddisk.
That suszane of 667MB took me about 2 to 3 secconds to load on SSD disk. So we realy cannot blame Blender here, but if it takes you 15 minutes upgrade your hardware, because your Disk IO speed is the bottle neck here.

If this is your regular job, then i stronlgy recommend you a SSD, this is not a bug in code; the waiting time you have is simply caused by the IO speed of your harddisk. That suszane of 667MB took me about 2 to 3 secconds to load on SSD disk. So we realy cannot blame Blender here, but if it takes you 15 minutes upgrade your hardware, because your Disk IO speed is the bottle neck here.
Author

I have a SSD and when I export the model to the SSD as .obj it takes several minutes. So that can not be the problem.

I don't want to overreach but I would like to point out that the file size of the blend file is 667 kilobytes, not 667 megabyte (if by "that Suzanne of 667MB" you mean my example file). So it would load very quickly, the reason being that the geometry is created at runtime and not saved in the file since it is generated by a subsurf modifier.

I have a SSD and when I export the model to the SSD as .obj it takes several minutes. So that can not be the problem. I don't want to overreach but I would like to point out that the file size of the blend file is 667 kilobytes, not 667 megabyte (if by "that Suzanne of 667MB" you mean my example file). So it would load very quickly, the reason being that the geometry is created at runtime and not saved in the file since it is generated by a subsurf modifier.

Added subscriber: @mano-wii

Added subscriber: @mano-wii

Changed status from 'Archived' to: 'Archived'

Changed status from 'Archived' to: 'Archived'

The code for exporting obj is written in python and python is usually slow.
By analyzing the export_obj.py code, there are some parts that could be improved using the numpy package.

One of the reasons for the slow code is the line with:
face_index_pairs.sort (key = sort_func)
That is called when the Keep Vertex Order option is checked.

Other reasons for slowness are the loops:

  • for v in me_verts:;
  • for f, f_index in face_index_pairs: (2x)
  • and possibly others, but I got tired of waiting;

Actually this slowness is quite inconvenient, but the code is not broken so the priority is not high.

Currently the developers are focusing their time on the Blender 2.8 Project. So I have no expectations that this problem will be corrected so soon.

In any case, it is good that it is taken into account.

The code for exporting obj is written in python and python is usually slow. By analyzing the `export_obj.py` code, there are some parts that could be improved using the numpy package. One of the reasons for the slow code is the line with: ```face_index_pairs.sort (key = sort_func)``` That is called when the `Keep Vertex Order` option is checked. Other reasons for slowness are the loops: - `for v in me_verts:`; - `for f, f_index in face_index_pairs:` (2x) - and possibly others, but I got tired of waiting; Actually this slowness is quite inconvenient, but the code is not broken so the priority is not high. Currently the developers are focusing their time on the `Blender 2.8 Project`. So I have no expectations that this problem will be corrected so soon. In any case, it is good that it is taken into account.
Member

Added subscriber: @LazyDodo

Added subscriber: @LazyDodo
Member

@PeterBoos as much fun as yelling at users is that they can't model and their hardware sucks, I decided to sit down and do some tests.

Decided to give it a whirl, exporting to an SSD drive.

testing procedure:

  1. Export from blender .obj which is a common format that virtually all applications know how to read

  2. Import into meshlab

  3. Import into blender

  4. Export from meshlab

With the file from this ticket as is (subdiv level 7):

exporting .obj from blender     : 5m05s (950mb file)
Meshlab export to .obj          : crashed couldn't test. 

Meshlab import of the .obj file : 1m24s 
Blender Import of the .obj file : had to kill it after 5 minutes after it consumed 18GB of ram

changed subdiv level to 5 and payed attention to ram usage:

exporting .obj from blender     : 18s (76mb file) 
Meshlab export to .obj          : 4s 

Meshlab import of the .obj file : 3s / 375mb ram
Blender Import of the .obj file : 18s / 1.3gb ram

changed subdiv level to 6 and payed attention to ram usage:

exporting .obj from blender     : 78s (243mb file) 
Meshlab export to .obj          : 22s  

Meshlab import of the .obj file : 22s / 2.0gb ram
Blender Import of the .obj file : 75s / 4.3gb ram

So in all instances blender was MUCH slower, and used MUCH MUCH more ram compared to other applications while performing the same tasks.

@PeterBoos i'm not sure if you just had a bad day or something else, but generally we try not to be this abusive to users, please try to be a tad more civil in the future?

@PeterBoos as much fun as yelling at users is that they can't model and their hardware sucks, I decided to sit down and do some tests. Decided to give it a whirl, exporting to an SSD drive. testing procedure: 1) Export from blender .obj which is a common format that virtually all applications know how to read 2) Import into meshlab 3) Import into blender 4) Export from meshlab With the file from this ticket as is (subdiv level 7): ``` exporting .obj from blender : 5m05s (950mb file) Meshlab export to .obj : crashed couldn't test. Meshlab import of the .obj file : 1m24s Blender Import of the .obj file : had to kill it after 5 minutes after it consumed 18GB of ram ``` changed subdiv level to 5 and payed attention to ram usage: ``` exporting .obj from blender : 18s (76mb file) Meshlab export to .obj : 4s Meshlab import of the .obj file : 3s / 375mb ram Blender Import of the .obj file : 18s / 1.3gb ram ``` changed subdiv level to 6 and payed attention to ram usage: ``` exporting .obj from blender : 78s (243mb file) Meshlab export to .obj : 22s Meshlab import of the .obj file : 22s / 2.0gb ram Blender Import of the .obj file : 75s / 4.3gb ram ``` So in all instances blender was *MUCH* slower, and used *MUCH* *MUCH* more ram compared to other applications while performing the same tasks. @PeterBoos i'm not sure if you just had a bad day or something else, but generally we try not to be this abusive to users, please try to be a tad more civil in the future?
Author

@LazyDodo @mano-wii Thank you for taking the issue seriously, the initial response was infuriating. It's sad that the issue won't be resolved soon but thank you for the hint with the vertex order, I'll keep that in mind :)

@LazyDodo @mano-wii Thank you for taking the issue seriously, the initial response was infuriating. It's sad that the issue won't be resolved soon but thank you for the hint with the vertex order, I'll keep that in mind :)

Added subscriber: @VukGardasevic

Added subscriber: @VukGardasevic

If you can, try exporting to Collada (dae) using the Collada (Default) exporter.
On a X3 455 (an old AMD 3 core):

  • it takes around 10-15 seconds for a subdivision 6 Suzanne (~ 2M polygons).
  • 57 seconds for a subdiv 7 Suzanne (~ 8M polygons) ( around 4 GB used in total as the subdivision was applied with the base of 2GB just for the model). The DAE file is 986 MB in size.
If you can, try exporting to Collada (dae) using the Collada (Default) exporter. On a X3 455 (an old AMD 3 core): - it takes around 10-15 seconds for a subdivision 6 Suzanne (~ 2M polygons). - 57 seconds for a subdiv 7 Suzanne (~ 8M polygons) ( around 4 GB used in total as the subdivision was applied with the base of 2GB just for the model). The DAE file is 986 MB in size.
Author

@VukGardasevic Thank you for the tip! My target platform (xnormal) even supports dae.

@VukGardasevic Thank you for the tip! My target platform (xnormal) even supports dae.

Changed status from 'Archived' to: 'Open'

Changed status from 'Archived' to: 'Open'

This comment was removed by @PeterBoos

*This comment was removed by @PeterBoos*

i am civil,
Let me give you some more numbers
Apply modifiers then save the file to a Blend file thats 667Mb of raw (but also compressed) blender data,
Without blend saving only applying the modifier, that raw data inside blender its about 1.7 Gig what you see on screen.
that's to be converted to obj since the formats are not the same it takes around 7Gig of data to be processed to become a obj file of more then 900Mb.
well it depends on the horsepower of your machine, over here its about 60 seconds.
And i'm kinda amazed that such a huge file is still processed that fast, for the amounts of data that it has to process.
Maybe add more Ram, or a faster processor (more cores) might help as well. (60 sec tested on Asus K95v a laptop 4 years old now).

So to say blender has a bug its slow, with these data numbers to me seams a rather bold statement.

Though its also subjective "there is no standard for fast" here, upgrading your hardware for sure would help.
You claim 15 minutes what i do in a minute, well i reopen because i only wonder what a python dev might think of this bold claim.
If you reconsider these facts and my earlier advice you might as well close it yourself.

Maybe as i'm not english my language was a bit to formal, i would advice people who start to use blender to go to blender artist.
People there are helping people out with problems like these, this however is a bug report area, not wishing area.
Essentially its working, and it depends on your hardware how fast it works.
And it depends on your workflow how much data you generate.

i am civil, Let me give you some more numbers Apply modifiers then save the file to a Blend file thats 667Mb of raw (but also compressed) blender data, Without blend saving only applying the modifier, that raw data inside blender its about 1.7 Gig what you see on screen. that's to be converted to obj since the formats are not the same it takes around 7Gig of data to be processed to become a obj file of more then 900Mb. well it depends on the horsepower of your machine, over here its about 60 seconds. And i'm kinda amazed that such a huge file is still processed that fast, for the amounts of data that it has to process. Maybe add more Ram, or a faster processor (more cores) might help as well. (60 sec tested on Asus K95v a laptop 4 years old now). So to say blender has a bug its slow, with these data numbers to me seams a rather bold statement. Though its also subjective "there is no standard for fast" here, upgrading your hardware for sure would help. You claim 15 minutes what i do in a minute, well i reopen because i only wonder what a python dev might think of this bold claim. If you reconsider these facts and my earlier advice you might as well close it yourself. Maybe as i'm not english my language was a bit to formal, i would advice people who start to use blender to go to blender artist. People there are helping people out with problems like these, this however is a bug report area, not wishing area. Essentially its working, and it depends on your hardware how fast it works. And it depends on your workflow how much data you generate.

Added subscriber: @ArtoKitula

Added subscriber: @ArtoKitula

@PeterBoos, There is no point adding more juice to the CPU when python export/import works on single core. Converting import/export scripts to multithreading would help a lot. That's why this is really valid ticket.

Let's start using! http:*assimp.sourceforge.net

@PeterBoos, There is no point adding more juice to the CPU when python export/import works on single core. Converting import/export scripts to multithreading would help a lot. That's why this is really valid ticket. Let's start using! [http:*assimp.sourceforge.net ](http:*assimp.sourceforge.net)
Member

Changed status from 'Open' to: 'Archived'

Changed status from 'Open' to: 'Archived'
Member

it's not a valid ticket, the import/export is working as designed, so technically this would be a feature request which doesn't belong on the tracker (so sadly i'm gonna have to abide by tracker rules and close the ticket).

That being said, there's nothing with how formal you were, but you straight up told an end user he didn't know how to do his job, no matter how you phrase that, that's not behavior that is appropriate on the tracker.

it's not a valid ticket, the import/export is working as designed, so technically this would be a feature request which doesn't belong on the tracker (so sadly i'm gonna have to abide by tracker rules and close the ticket). That being said, there's nothing with how formal you were, but you straight up told an end user he didn't know how to do his job, no matter how you phrase that, that's not behavior that is appropriate on the tracker.

ah i see now, at first i responded quickly, i wasnt judging him, but the line was a general statement.
i should have written that poor sculpts, result in poor meshes, i was trying to explain that in general there (like a saying), but not to judge his specific level.

My excuses to him as, i had no intend to blame him there, bad tutor choice of words from my side.

ah i see now, at first i responded quickly, i wasnt judging him, but the line was a general statement. i should have written that poor sculpts, result in poor meshes, i was trying to explain that in general there (like a saying), but not to judge his specific level. My excuses to him as, i had no intend to blame him there, bad tutor choice of words from my side.

Added subscriber: @JacobMerrill-1

Added subscriber: @JacobMerrill-1

yeah I am using this right now on a production file and it takes ages
almost as long as baking a 4k

yeah I am using this right now on a production file and it takes ages almost as long as baking a 4k
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
7 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#53236
No description provided.