The OBJ parser was primarily using StringRef for convenience, with
functions like "skip whitespace" or "parse a number" taking an input
stringref, representing an input line, and returning a new stringref,
representing the remainder of the line. This is convenient, but does
more work than strictly needed -- while parsing, only the "beginning"
of the line ever changes by moving forward; the end of the line
always stays the same. We can change the code to take a pair of
pointers (begin of line, end of line) as input, and make the
functions return the new begin of line pointer. This makes the return
value neatly fit into a processor register, which StringRef did not.
On Windows, this does result in non-trivial speedups in the actual
OBJ file parsing part, due to Windows calling convention where return
values larger than 64 bits are returned via memory. Does not
measurably affect performance on Mac/Linux, because the calling
convention there uses a pair of 64-bit registers to return a
StringRef.
End-to-end times of importing several test files, on Windows
(VS2022 build, Ryzen 5950X):
- Monkey subdivided to level 6, no normals (220MB file): 1.25s -> 0.85s
- Rungholt minecraft level (270MB file): 7.0s -> 5.8s
- Blender 3 splash scene (2.4GB file): 49.1s -> 45.5s
The full import process has a lot of other overhead besides actual
OBJ file parsing (mostly creating actual blender objects out of
parsed data). In pure parsing, in the monkey test scene above, the
parsing part goes 1.0s -> 0.6s.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14936
New OBJ exporter is missing "Path Mode" setting for exporting .mtl
files. The options that used to be available were: Auto, Absolute,
Relative, Match, Strip Path, Copy. All of them are important. The new
behavior (without any UI option to control it) curiously does not match
any of the previous setting. New behavior is like "Relative, but to the
source blender file, and not the destination export file".
Most of the previous logic was only present in Python based code
(bpy_extras.io_utils.path_reference and friends). The bulk of this
commit is porting that to C++.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14906
While possible extra whitespace after all OBJ/MTL keywords was properly
skipped, it was not done for the "f" (face definition) keyword.
While at it, also support indented keywords, i.e. extra whitespace at
the beginning of the line.
There's a tiny bit of performance drop while importing (e.g. importing
blender 3.0 splash scene: 53.38sec -> 54.21sec on my machine). But
correctness is more important.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14854
Fixes T97794 (which is a reintroduction of an older issue T67266 that
has been fixed in the python importer, but the fix was not in the C++
one). Some software produces OBJ files with mtllib statements like
mtllib "file name in quotes.mtl", and the new importer was not stripping
the quotes away.
While at it, I noticed that MTLParser constructor was taking a StringRef
and treating it as a zero-terminated string, which is not necessarily
the case. Fixed that by explicitly using a StringRefNull type.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14838
- Fix T97793: when a UV coordinate after vt is missing, use zero. While
at it, also use zeroes for positions & normals, since "maximum
possible float" is very likely to cause issues in imported meshes.
- Fix T97795: use 1.0 default if -bm value is missing, instead of zero.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14826
Fix several correctness issues where the new OBJ/MTL importer was not
producing the same results as the old one, mostly because the code for
some reason had slightly different logic. Fixes T97757:
- When .obj file tries to use a material that does not exist, the code
was continuing to use the previous material, instead of creating new
default one, as the previous importer did.
- Previous importer was always searching/parsing "foo.mtl" for a
"foo.obj" file, even if the file itself does not contain
"mtllib foo.mtl" statement. One file from T97757 repros happens to
depend on that, so resurrect that behavior.
- When IOR (Ni) or Alpha (d) are not specified in .mtl file, do not
wrongly set -1 values to the blender material.
- When base (Kd) or emissive (Ke) colors are not specified in the .mtl
file, do not set them on the blender material.
- Roughness and metallic values used by viewport shading were not set
onto blender material.
- The logic for when metallic was set to zero was incorrect; it should
be set to zero when "not using reflection", not when "mtl file does
not contain metallic".
- Do not produce a warning when illum value is not spelled out in .mtl
file, treat as default (1).
- Parse illum as a float just like python importer does, as to not
reintroduce part of T60135.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14822
The old python importer had a "if do_transparency, set blend_method to
BLEND" type of logic. This bit was missing in the new importer; it was
only setting the eevee blend method when a transparency texture was
present, but not in other cases of transparency (as driven by MTL
"illum" mode).
Reviewd By: Howard Trickey
Differential Revision: https://developer.blender.org/D14783
Even if available OBJ/MTL format documentations don't explicitly specify
which characters can possibly separate keywords & arguments, turns out
some files out there in the wild use TAB character after the line
keywords. Which is something the new 3.2 importer was not quite
expecting (T97417).
Fix this by factoring out a utility function that checks if line starts
with a keyword followed by any whitespace, and using that across the
importer. Also fix some other "possible whitespace around name-like
parts" of obj/mtl parser as pointed out by the repro files in T97417.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14782
Continued improvements to the new C++ based OBJ importer.
Performance: about 2x faster.
- Rungholt.obj (several meshes, 263MB file): Windows 12.7s -> 5.9s, Mac 7.7s -> 3.1s.
- Blender 3.0 splash (24k meshes, 2.4GB file): Windows 97.3s -> 53.6s, Mac 137.3s -> 80.0s.
- "Windows" is VS2022, AMD Ryzen 5950X (32 threads), "Mac" is Xcode/clang 13, M1Max (10 threads).
- Slightly reduced memory usage during import as well.
The performance gains are a combination of several things:
- Replacing `std::stof` / `std::stoi` with C++17 `from_chars`.
- Stop reading input file char-by-char using `std::getline`, and instead read in 64kb chunks, and parse from there (taking care of possibly handling lines split mid-way due to chunk boundaries).
- Removing abstractions for splitting a line by some char,
- Avoid tiny memory allocations: instead of storing a vector of polygon corners in each face, store all the corners in one big array, and per-face only store indices "where do corners start, and how many". Likewise, don't store full string names of material/group names for each face; only store indices into overall material/group names arrays.
- Stop always doing mesh validation, which is slow. Do it just like the Alembic importer does: only do validation if found some invalid faces during import, or if requested by the user via an import setting checkbox (which defaults to off).
- Stop doing "collection sync" for each object being added; instead do the collection sync right after creating all the objects.
Cleanup / Robustness:
This reworking of parser (see "removing abstractions" point above) means that all the functions that were in `parser_string_utils` file are gone, and replaced with different set of functions. However they are not OBJ specific, so as pointed out during review of the previous differential, they are now in `source/blender/io/common` library.
Added gtest coverage for said functions as well; something that was only indirectly covered by obj tests previously.
Rework of some bits of parsing made the parser actually better able to deal with invalid syntax. E.g. previously, if a face corner were a `/123` string, it would have incorrectly treated that as a vertex index (since it would get "hey that's one number" after splitting a string by a slash), instead of properly marking it as invalid syntax.
Added gtest coverage for .mtl parsing; something that was not covered by any tests at all previously.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14586
- Was not exporting "Poly" curves at all,
- Had a crash when a single object contains multiple curves of different types -- it had a check for "is this nurbs compatible?" only for the first curve, and then proceeded to treat the other curves as nurbs as well, without checking for validity.
Fixed both issues by doing the same logic as in the old python exporter:
- Poly curves are supported,
- Treat object as "nurbs compatible" only if all the curves within it are nurbs compatible.
Added test coverage in the gtest suite. While at it, made "all_curves" test use the "golden obj file template" style test, instead of a manually coded test that checks intermediate objects but does not check the final exported result.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14611
The new 3.1 OBJ exporter code had incorrect code to determine which vertex group a polygon belongs to -- for each vertex, it was only looking at the first vertex group it has, and not using the group weight either.
This 99% fixes T96824, but not 100% on the user's submitted mesh -- exactly two faces from that mesh get assigned a different group compared to the old exporter. Either choice is "correct" given that on these two faces there are two vertex groups with equal contribution. The old Python exporter was picking the group based on internal python group name map order, whereas the new C++ exporter is picking the group with the lowest index, in case of ties. I'm not sure if it's possible to fix this TBH, will have to wait until the importer is also C++.
While at it, the new vertex group calculation code was doing a lot of redundant work for each and every face (traversing group lists several times, allocating & freeing memory), so I fixed that. Exporting a 6-level subdivided Monkey mesh with 30 vertex groups was taking 810ms, now takes 330ms.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14500
The all_objects.blend test scene (in subversion tests repo) contained an
object with a subdivision surface. Which changes vertex positions
slightly, depending on used OpenSubDiv version and the compile flags. It
seems that the intent of the test was "test export of meshes that use
modifiers", so I changed that object to be a cube with a simple "taper"
modifier instead.
While at it, changed OBJ exporter test code to always print the
"expected and what we got" text difference details, when a test fails.
Much easier to see than just "the files are different" output. The code
to print that was behind an off by default flag for some reason.
This diff should get comitted together with updated all_objects templates
in subversion tests repo.
Reviewed By: Sebastian Parborg
Differential Revision: https://developer.blender.org/D14597
This commit furthers some of the changes that were started in
rBb9febb54a492 and subsequent commits by changing the way surface
objects are presented to render engines and other users of evaluated
objects in the same way. Instead of presenting evaluated surface objects
as an `OB_SURF` object with an evaluated mesh, `OB_SURF` objects
can now have an evaluated geometry set, which uses the same system
as other object types to deal with multi-type evaluated data.
This clarification makes it more obvious that lots of code that dealt
with the `DispList` type isn't used. It wasn't before either, now it's
just *by design*. Over 1100 lines can be removed. The legacy curve
draw cache code is much simpler now too. The idea behind the further
removal of `DispList` is that it's better to focus optimization efforts
on a single mesh data structure.
One expected functional change is that the evaluated mesh from surface
objects can now be used in geometry nodes with the object info node.
Cycles and the OBJ IO tests had to be tweaked to avoid using evaluated
surface objects instead of the newly exposed mesh objects.
Differential Revision: https://developer.blender.org/D14550
This takes state of soc-2020-io-performance branch as it was at
e9bbfd0c8c (2021 Oct 31), merges latest master (2022 Apr 4),
adds a bunch of tests, and fixes a bunch of stuff found by said
tests. The fixes are detailed in the differential.
Timings on my machine (Windows, VS2022 release build, AMD Ryzen
5950X 32 threads):
- Rungholt minecraft level (269MB file, 1 mesh): 54.2s -> 14.2s
(memory usage: 7.0GB -> 1.9GB).
- Blender 3.0 splash scene: "I waited for 90 minutes and gave up"
-> 109s. Now, this time is not great, but at least 20% of the
time is spent assigning unique names for the imported objects
(the scene has 24 thousand objects). This is not specific to obj
importer, but rather a general issue across blender overall.
Test suite file updates done in Subversion tests repository.
Reviewed By: @howardt, @sybren
Differential Revision: https://developer.blender.org/D13958
Original report (T96763) only reported the issue of double-space before the texture path, but while adding test coverage I found some other issues that I fixed while at it:
- Incorrectly emits two spaces between `map_Xx` keyword and the texture path, leading to some 3rd party software not finding the textures,
- Emissive texture map (`map_Ke`) was not exported,
- When Mapping node is used on the texture UVs, the "Location" and "Scale" values were mixed up (location written as "scale", scale written as "location).
Added gtest coverage.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14519
On Windows/MSVC this gives a minor (~20%) speedup presumably due to a faster float/int formatter. On macOS (Xcode13), this gives a massive speedup, since snprintf that is in system libraries ends up spending almost all the time inside some locale-related mutex lock.
The actual exporter code becomes quite a bit smaller too, since it does not have to do any juggling to support std::string arguments, and the buffer handling code is smaller as well.
Windows (VS2022 release build, Ryzen 5950X 32 threads) timings:
- Blender 3.0 splash scene (2.4GB obj): 4.57s -> 3.86s
- Monkey subdivided level 6 (330MB obj): 1.10s -> 0.99s
macOS (Xcode 13 release build, Apple M1Max) timings:
- Blender 3.0 splash scene (2.4GB obj): 21.03s -> 5.52s
- Monkey subdivided level 6 (330MB obj): 3.28s -> 1.20s
Linux (ThreadRipper 3960X 48 threads) timings:
- Blender 3.0 splash scene (2.4GB obj): 10.10s -> 4.40s
- Monkey subdivided level 6 (330MB obj): 2.16s -> 1.37s
The produced obj/mtl files are identical to before.
Reviewed By: Howard Trickey, Dalai Felinto
Differential Revision: https://developer.blender.org/D13998
This change makes it possible to add implementation of common
C++ methods for DNA structures which helps ensuring unsafe
operations like shallow copy are done explicitly.
For example, creating a shallow copy used to be:
Object temp_object = *input_object;
In the C++ context it was seen like the temp_object is
properly decoupled from the input object, while in the
reality is it not. Now this code becomes:
Object temp_object = blender::dna::shallow_copy(*input_object);
The copy and move constructor and assignment operators are
now explicitly disabled.
Other than a more explicit resource management this change
also solves a lot of warnings generated by the implicitly
defined copy constructors w.r.t dealing with deprecated fields.
These warnings were generated by Apple Clang when a shallow
object copy was created via implicitly defined copy constructor.
In order to enable C++ methods for DNA structures a newly added
macro `DNA_DEFINE_CXX_METHODS()` is to be used:
tpyedef struct Object {
DNA_DEFINE_CXX_METHODS(Object)
...
} Object;
For the shallow copy use `blender::dna::shallow_copy()`.
The implementation of the memcpy is hidden via an internal DNA
function to avoid pulling `string.h` into every DNA header.
This means that the solution does not affect on the headers
dependencies.
---
Ideally `DNA_shallow_copy` would be defined in a more explicit
header, but don;t think we have a suitable one already. Maybe
we can introduce `DNA_access.h` ?
Differential Revision: https://developer.blender.org/D14427
This reverts commit 8c44793228.
Apparently, this generated a lot of warnings in GCC.
Didn't find a quick solution and is it not something I want to be
trading between (more quiet Clang in an expense of less quiet GCC).
Will re-iterate on the patch are re-commit it.
This change makes it possible to add implementation of common
C++ methods for DNA structures which helps ensuring unsafe
operations like shallow copy are done explicitly.
For example, creating a shallow copy used to be:
Object temp_object = *input_object;
In the C++ context it was seen like the temp_object is
properly decoupled from the input object, while in the
reality is it not. Now this code becomes:
Object temp_object = blender::dna::shallow_copy(*input_object);
The copy and move constructor and assignment operators are
now explicitly disabled.
Other than a more explicit resource management this change
also solves a lot of warnings generated by the implicitly
defined copy constructors w.r.t dealing with deprecated fields.
These warnings were generated by Apple Clang when a shallow
object copy was created via implicitly defined copy constructor.
In order to enable C++ methods for DNA structures a newly added
macro `DNA_DEFINE_CXX_METHODS()` is to be used:
tpyedef struct Object {
DNA_DEFINE_CXX_METHODS(Object)
...
} Object;
For the shallow copy use `blender::dna::shallow_copy()`.
The implementation of the memcpy is hidden via an internal DNA
function to avoid pulling `string.h` into every DNA header.
This means that the solution does not affect on the headers
dependencies.
---
Ideally `DNA_shallow_copy` would be defined in a more explicit
header, but don;t think we have a suitable one already. Maybe
we can introduce `DNA_access.h` ?
Differential Revision: https://developer.blender.org/D14427
Currently there is a "calc_face_normal" argument to mesh to bmesh
conversion, but vertex normals had always implicitly inherited whatever
dirty state the mesh input's vertex normals were in. Probably they were
most often assumed to not be dirty, but this was never really correct in
the general case.
Ever since the refactor to move vertex normals out of mesh vertices,
cfa53e0fbe, the copying logic has been explicit: copy the
normals when they are not dirty. But it turns out that more control is
needed, and sometimes normals should be calculated for the resulting
BMesh.
This commit adds an option to the conversion to calculate vertex
normals, true by default. In almost all places except the decimate
and edge split modifiers, I just copied the value of the
"calc_face_normals" argument.
Differential Revision: https://developer.blender.org/D14406
Old python exporter in 3.0 and earlier ordered faces by material,
but the new C++ exporter in 3.1+ did not, and was just writing them
in whatever is the order of the mesh data structure.
This mostly does not cause problems, except in some apps e.g.
Procreate -- for large enough meshes, this lack of
"order by material" (which ends up having more usemtl lines)
ends up creating more mesh subsets than necessary inside Procreate.
The change is not computationally heavy, e.g. exporting 6-level
subdivided Monkey mesh goes 1085ms -> 1105ms on my machine.
Reviewed By: @howardt
Differential Revision: https://developer.blender.org/D14368
This is patch D14349 from Aras Pranckevicius.
The logic in the code was _completely different_ from the documentation
and what the python exporter in 3.0 did. The new code assumed that
"export material groups" meant "append material name to the object name",
and was only ever kicking in when the "export object groups" option was
also checked. But the proper behavior (as in 3.0 exporter & the online docs),
is to emit g objectname_materialname before each usemtl line. Which is something entirely else.
This is patch D14347 from Aras Pranckevicius.
Instead of scaling "the scene" (i.e. transform vertices by object matrix,
then multiply by scale factor), it was instead first applying the scale
factor in local space, and then transforming by the object matrix.
This applies patch D14343 from Aras Pranckevicius, with a description:
The new 3.1+ OBJ exporter did not have correct logic when faced with
non-uniform & mirrored (negative on odd number of axes) object scale:
- Normals were not transformed correctly (should use inverse transpose of the matrix),
and were not normalized,
- Face order was not "flipped" when transform has negative scale on odd number of axes
(visible when using "face orientation" viewport overlay).
This commit renames enums related the "Curve" object type and ID type
to add `_LEGACY` to the end. The idea is to make our aspirations clearer
in the code and to avoid ambiguities between `CURVE` and `CURVES`.
Ref T95355
To summarize for the record, the plans are:
- In the short/medium term, replace the `Curve` object data type with
`Curves`
- In the longer term (no immediate plans), use a proper data block for
3D text and surfaces.
Differential Revision: https://developer.blender.org/D14114
This patch reverses the dependency between `BLI_math_vec_types.hh` and
`BLI_math_vector.hh`. Now the higher level `blender::math` functions
depend on the header that defines the types they work with, rather than
the other way around.
The initial goal was to allow defining an `enable_if` in the types header
and using it in the math header. But I also think this operations to types
dependency is more natural anyway.
This required changing the includes some files used from the type
header to the math implementation header. I took that change a bit
further removing the C vector math header from the C++ header;
I think that helps to make the transition between the two systems
clearer.
Differential Revision: https://developer.blender.org/D14112
