blender
/
blender
132
398
Fork 577
Code Issues 6.2k Pull Requests 745 Projects 19 Wiki Activity

Export material to MaterialX for Hydra render #111765

Manually merged
Brecht Van Lommel merged 48 commits from DagerD/blender:matx-export-material into main 2023-09-26 18:56:40 +02:00
Conversation 18 Commits 48 Files Changed 79 +4062 -19
Bogdan Nagirniak commented 2023-09-01 00:27:11 +02:00
Contributor
Copy Link

Purpose

Significant part of Hydra render engine integration from #104712 is export Blender's material nodegraph to MaterialX nodegraph which automatically can be translated to USD shaders and use by Hydra render delegates.

Done

  1. Created node parsing system in bf_nodes_shader project. Placed into source/blender/nodes/shader/materialx folder. It consists:

    • Base class NodeParser which provides required functionality for walking through node tree and translating blender node into MaterialX.
    • Inherited classes (every class corresponds to one blender node) should override compute() method.
    • NodeParser operates with objects of class NodeItem which simplifies work with MaterialX nodes and values: implements arithmetic functions and other math functions, simplifies work with node inputs and outputs.

  2. Added bl_use_materialx to Render engine property. When it is set, MaterialX is utilized for exporting materials to Hydra. Code moved from #111219.

  3. Already implemented export of following blender nodes: BSDFPrincipled, Invert, Math, TexChecker, TexEnvironment, TexImage, TexNoise.

  4. Added CLOG logger to node parsing system.

TODOs

  • Improve NodeItem to use enums instead string constants.
  • Export of other blender nodes as much as possible. This task would be splitted to subtasks and updated here.
  • Working with group nodes.
### Purpose Significant part of Hydra render engine integration from #104712 is export Blender's material nodegraph to MaterialX nodegraph which automatically can be translated to USD shaders and use by Hydra render delegates. ### Done 1. Created node parsing system in `bf_nodes_shader` project. Placed into `source/blender/nodes/shader/materialx` folder. It consists: - Base `class NodeParser` which provides required functionality for walking through node tree and translating blender node into MaterialX. - Inherited classes (every class corresponds to one blender node) should override `compute()` method. - `NodeParser` operates with objects of `class NodeItem` which simplifies work with MaterialX nodes and values: implements arithmetic functions and other math functions, simplifies work with node inputs and outputs. 2. Added `bl_use_materialx` to Render engine property. When it is set, MaterialX is utilized for exporting materials to Hydra. Code moved from #111219. 3. Already implemented export of following blender nodes: BSDFPrincipled, Invert, Math, TexChecker, TexEnvironment, TexImage, TexNoise. 4. Added `CLOG` logger to node parsing system. ### TODOs - [x] Improve `NodeItem` to use enums instead string constants. - [x] Export of other blender nodes as much as possible. This task would be splitted to subtasks and updated here. - [x] Working with group nodes.
🚀 2
Bogdan Nagirniak added 10 commits 2023-09-01 00:27:26 +02:00
b89fb0785e Add basic supported nodes 
### Purpose
Add basic supported nodes: `Material output`, `Principled BSDF`, `Image Texture`.

### Technical steps
* Added base class `MaterialXNode` for all MatX nodes
* Added basic supported nodes: `Material output`, `Principled BSDF`, `Image Texture`
* Added function `export_nodegraph`
* Fix typo

Pull Request: DagerD/blender#1
6822cfe948 Add bl_use_materialx property 
### Purpose
Add `bl_use_materialx` to `RenderEngine`. MaterialX is utilized for exporting materials to Hydra.

### Technical steps
Added property `bl_use_materialx` to `RenderEngine`.

Co-authored-by: Vasyl-Pidhirskyi <vpidhirskyi@gmail.com>
Pull Request: DagerD/blender#3
c5b48921bf Create parsing system that converts supported nodes and ignores unsupported 
Implemented node export system: created `class NodeParser` and `class NodeItem`.

Co-authored-by: Bogdan Nagirniak <bodyan@gmail.com>
Pull Request: DagerD/blender#2
2a2c90226f Arithmetic support for NodeItem and node implementation with arithmetic 
### Purpose
NodeItem requires arithmetic functions for export implementation of blender nodes.

### Technical steps
1. Added basic arithmetic functions to `NodeItem`.
2. Added some math nodes: `InvertNodeParser`, `MathNodeParser`, `MixRGBNodeParser`.
3. Improved parser declaration: added `DECLARE_PARSER` macros.

### Note
Only `InvertNodeParser` is implemented. `MathNodeParser`, `MixRGBNodeParser` are in TODOs.

Pull Request: DagerD/blender#4
6e49ba3d76 Implement export of Math node. Continue other arithmetic support for NodeItem 
### Purpose
Continuing implementation of arithmetic functions for `NodeItem`. Implement Math node.

### Technical steps
1. Separated implementation of `NodeItem` and `NodeParser` to separate files.
2. Added more functions to `NodeItem`.
3. Implemented Blender Math node.

### Notes
Some operations from Math node aren't implemented. Probably will be done in separate task.

Pull Request: DagerD/blender#6
4dfb0c5857 Export to MatX various Texture nodes 
### Purpose
Add support for various Texture nodes.

### Technical steps
Added support for:
* Checker Texture
* Noise Texture
Extended NodeItem::set_input() by adding output_name. Added NodeItem::add_output()

### Notes
Environment Texture. `Projection` property is not supported for now.

Co-authored-by: Bogdan Nagirniak <bodyan@gmail.com>
Pull Request: DagerD/blender#5
fa0aea502f MaterialX logging 
### Purpose
Change debug printf to CLOG logger. Add other logging.

### Technical steps
1. Applied CLOG logger to `bf_nodes_shader` project.
2. Changed debug `printf` to logging. Added other logging.
3. Fixed "Use MaterialX" in `rna_render.cc`.

### Notes
Don't use `--log-show-basename` blender running flag in Windows, because of clog crash due to incorrect path separator for `bf_nodes_shader` project.

Pull Request: DagerD/blender#7
Bogdan Nagirniak requested review from Brecht Van Lommel 2023-09-01 00:28:18 +02:00
Bogdan Nagirniak requested review from Brian Savery (AMD) 2023-09-01 00:28:18 +02:00
Bogdan Nagirniak requested review from Georgiy Markelov 2023-09-01 00:28:18 +02:00
Bogdan Nagirniak requested review from Vasyl Pidhirskyi 2023-09-01 00:28:18 +02:00
Brian Savery (AMD) requested changes 2023-09-01 18:37:26 +02:00
Brian Savery (AMD) left a comment
Member
Copy Link

See my comments about not converting parts of the node tree to materialx. There are many places this happens, please change.

See my comments about not converting parts of the node tree to materialx. There are many places this happens, please change.
source/blender/nodes/shader/materialx/nodes/bsdf_principled.cc Outdated
@ -0,0 +17,4 @@
NodeItem subsurface = get_input_value("Subsurface");
NodeItem subsurface_radius = empty();
NodeItem subsurface_color = empty();
if (subsurface != zero) {
Brian Savery (AMD) commented 2023-09-01 18:36:01 +02:00
Member
Copy Link

I would argue here.... for completeness that we should export the full node tree, even if some parts of it are "inactive".

E.G. Here we only convert the part of the tree with Subsurface Color input if subsurface is turned on. One could imagine that someone wants to take that exported materialx and then turn on the subsurface, then the part for the subsurface color is missing.

It's up to the MaterialX code generation and compiler to prune parts of the node tree that aren't needed. Not here.

I would argue here.... for completeness that we should export the full node tree, even if some parts of it are "inactive". E.G. Here we only convert the part of the tree with Subsurface Color input if subsurface is turned on. One could imagine that someone wants to take that exported materialx and then turn on the subsurface, then the part for the subsurface color is missing. It's up to the MaterialX code generation and compiler to prune parts of the node tree that aren't needed. Not here.
👍 1
Bogdan Nagirniak commented 2023-09-08 10:11:16 +02:00
Author
Contributor
Copy Link

This is done

This is done
BogdanNagirniak marked this conversation as resolved
Bogdan Nagirniak added 1 commit 2023-09-07 11:22:50 +02:00
a464b5854e Implement export of Shader BSDF nodes 
### Purpose
Shader nodes has to be implemented. Adjust parsing system for export shaders.

### Technical steps
1. Added supported shader types to `NodeItem::Type`, added `Type::Any`, added comments.
2. Created `class ShaderNodeParser` special for parse shaders.
3. Added type conversion to `get_input_link()` and `get_input_default()`.
4. Added file templates for export new shaders. Implemented `AddShader`, `BSDFDiffuse`, `Emission`, `MixShader`.
5. Improved TexChecker using arithmetic functions.

### Notes
Other shaders aren't implemented, added TODOs. They'll be implemented in next PRs.

Pull Request: DagerD/blender#13
Bogdan Nagirniak added 1 commit 2023-09-07 15:10:22 +02:00
a610ec2ba3 Parsing Improvements 
### Purpose
Perform parsing Improvements

### Technical steps
* Added `NodeItem::create_node()`
* Added `NodeItem::set_input_output()`
* Removed `NodeItem::set_input()` with `out_name` parameter
* `NodeParser::create_node()`: changed type to `NodeItem::Type`
* Logging improvements
* `NodeParser::value()` -> `NodeParser::val()`
* Renamings and code improvements

Pull Request: DagerD/blender#14
Brecht Van Lommel requested changes 2023-09-07 18:45:56 +02:00
Brecht Van Lommel left a comment
Owner
Copy Link

Overall seems quite straightforward to understand.

Note sure how hard it would be to move the MaterialX export code into the existing shader node files, but I think it's important. I think it should be possible without too many changes to the overall design. Worst case you could have some begin/end macros that do the class and callback function definition, with the compute implementation between the two.

Overall seems quite straightforward to understand. Note sure how hard it would be to move the MaterialX export code into the existing shader node files, but I think it's important. I think it should be possible without too many changes to the overall design. Worst case you could have some begin/end macros that do the class and callback function definition, with the compute implementation between the two.
source/blender/io/usd/CMakeLists.txt Outdated
@ -210,2 +210,4 @@
endif()
if(WITH_MATERIALX)
list(APPEND LIB MaterialXCore)
Brecht Van Lommel commented 2023-09-07 17:32:27 +02:00
Owner
Copy Link

Add add_definitions(-DWITH_MATERIALX)

Add `add_definitions(-DWITH_MATERIALX)`
BogdanNagirniak marked this conversation as resolved
source/blender/io/usd/hydra/material.cc Outdated
@ -12,6 +12,9 @@
#include <pxr/imaging/hd/tokens.h>
#include <pxr/usdImaging/usdImaging/materialParamUtils.h>
#include <pxr/usd/usdMtlx/reader.h>
Brecht Van Lommel commented 2023-09-07 17:33:04 +02:00
Owner
Copy Link

Add #ifdef WITH_MATERIALX here and other places in this file, to make it possible to build without materialx.

Add `#ifdef WITH_MATERIALX` here and other places in this file, to make it possible to build without materialx.
BogdanNagirniak marked this conversation as resolved
source/blender/nodes/shader/CMakeLists.txt Outdated
@ -24,0 +24,4 @@
if(WITH_HYDRA)
list(APPEND INC
../../io/usd
../../../../intern/clog
Brecht Van Lommel commented 2023-09-07 17:35:21 +02:00
Owner
Copy Link

Add clog the general includes, not need to have this conditional.

Add clog the general includes, not need to have this conditional.
BogdanNagirniak marked this conversation as resolved
source/blender/nodes/shader/CMakeLists.txt Outdated
@ -139,0 +175,4 @@
materialx/nodes/tex_environment.cc
materialx/nodes/tex_image.cc
materialx/nodes/tex_noise.cc
materialx/nodes/vector_math.cc
Brecht Van Lommel commented 2023-09-07 17:40:07 +02:00
Owner
Copy Link

Can the code for all these node types move into the corresponding source/blender/nodes/shader/nodes/node_shader_*.cc file? We are trying to make node implementations self contained instead of spread across multiple files.

Otherwise it's likely developers will forget to update the materialx code when making changes.

A new callback would be added to bNodeType, similar to gpu_fn, that would convert the node to materialx.

Can the code for all these node types move into the corresponding `source/blender/nodes/shader/nodes/node_shader_*.cc` file? We are trying to make node implementations self contained instead of spread across multiple files. Otherwise it's likely developers will forget to update the materialx code when making changes. A new callback would be added to `bNodeType`, similar to `gpu_fn`, that would convert the node to materialx.
BogdanNagirniak marked this conversation as resolved
source/blender/nodes/shader/materialx/material.cc Outdated
@ -0,0 +23,4 @@
MaterialX::DocumentPtr doc = MaterialX::createDocument();
if (material->use_nodes) {
material->nodetree->ensure_topology_cache();
Brecht Van Lommel commented 2023-09-07 18:28:36 +02:00
Owner
Copy Link

This should create a temporary copy of the node tree with ntreeLocalize, similar as is done for generating the GLSL shader. That will resolve any muting and reroute nodes, as handling them as part of the export process is not easy.

Node groups might ideally be preserved in export, but it would be easiest to start with exporting them expanded. For this it would be easiest to use the functions also called from ntreeGPUMaterialNodes:

  ntree_shader_groups_remove_muted_links(localtree);
  ntree_shader_groups_expand_inputs(localtree);
  ntree_shader_groups_flatten(localtree);
This should create a temporary copy of the node tree with `ntreeLocalize`, similar as is done for generating the GLSL shader. That will resolve any muting and reroute nodes, as handling them as part of the export process is not easy. Node groups might ideally be preserved in export, but it would be easiest to start with exporting them expanded. For this it would be easiest to use the functions also called from `ntreeGPUMaterialNodes`: ``` ntree_shader_groups_remove_muted_links(localtree); ntree_shader_groups_expand_inputs(localtree); ntree_shader_groups_flatten(localtree); ```
Bogdan Nagirniak commented 2023-09-19 09:03:42 +02:00
Author
Contributor
Copy Link

Exporting of group nodes is implemented in materialx/group_nodes.cc/.h

Exporting of group nodes is implemented in `materialx/group_nodes.cc/.h`
BogdanNagirniak marked this conversation as resolved
source/blender/nodes/shader/materialx/nodes/add_shader.cc Outdated
@ -0,0 +29,4 @@
case NodeItem::Type::SurfaceShader: {
res = get_input_shader(0, shader_type_);
if (!res) {
res = get_input_shader(1, shader_type_);
Brecht Van Lommel commented 2023-09-07 18:20:24 +02:00
Owner
Copy Link

Can you add a comment explaining this logic, why it picks one or the other instead of adding? Is this something that can be improved?

Can you add a comment explaining this logic, why it picks one or the other instead of adding? Is this something that can be improved?
BogdanNagirniak marked this conversation as resolved
source/blender/nodes/shader/materialx/nodes/mix_shader.cc Outdated
@ -0,0 +31,4 @@
break;
}
case NodeItem::Type::SurfaceShader: {
res = get_input_shader(1, NodeItem::Type::SurfaceShader);
Brecht Van Lommel commented 2023-09-07 18:32:26 +02:00
Owner
Copy Link

Similar questions as add shader, not sure what this is doing.

Similar questions as add shader, not sure what this is doing.
BogdanNagirniak marked this conversation as resolved
source/blender/nodes/shader/materialx/nodes/node_item.h Outdated
@ -0,0 +8,4 @@
namespace blender::nodes::materialx {
class NodeItem {
Brecht Van Lommel commented 2023-09-07 18:18:54 +02:00
Owner
Copy Link

Can you add a comment explaining what this class is?

Can you add a comment explaining what this class is?
BogdanNagirniak marked this conversation as resolved
source/blender/nodes/shader/materialx/nodes/node_item.h Outdated
@ -0,0 +51,4 @@
static std::string type(Type type);
/* Operators */
operator bool() const;
Brecht Van Lommel commented 2023-09-07 18:24:11 +02:00
Owner
Copy Link

Can you change this to bool is_empty() const? I would only use operator overloading when the meaning is more clear.

Can you change this to `bool is_empty() const`? I would only use operator overloading when the meaning is more clear.
Bogdan Nagirniak commented 2023-09-13 11:13:27 +02:00
Author
Contributor
Copy Link

Such bool operator is used during the code in several places, for example:

  NodeItem normal = get_input_link("Normal", NodeItem::Type::Vector3);
....
  if (normal) {
    res.set_input("normal", normal);
  }

And seems for me it suits there very nice. Are you sure using is_empty() would be better?

Such bool operator is used during the code in several places, for example: ``` NodeItem normal = get_input_link("Normal", NodeItem::Type::Vector3); .... if (normal) { res.set_input("normal", normal); } ``` And seems for me it suits there very nice. Are you sure using `is_empty()` would be better?
Brecht Van Lommel commented 2023-09-14 17:05:55 +02:00
Owner
Copy Link

Ok, we can keep it as is since it's a bit like a null pointer check.

Ok, we can keep it as is since it's a bit like a null pointer check.
brecht marked this conversation as resolved
source/blender/nodes/shader/materialx/nodes/node_parser.cc Outdated
@ -0,0 +136,4 @@
break;
switch (from_node->typeinfo->type) {
CASE_NODE_TYPE(SH_NODE_BRIGHTCONTRAST, BrightContrastNodeParser)
Brecht Van Lommel commented 2023-09-07 18:25:08 +02:00
Owner
Copy Link

If there is a node type callback, this switch statement will not be needed anymore.

If there is a node type callback, this switch statement will not be needed anymore.
BogdanNagirniak marked this conversation as resolved
source/blender/nodes/shader/materialx/nodes/tex_image.cc Outdated
@ -0,0 +20,4 @@
/* TODO: What if Blender built without Hydra? Also io::hydra::cache_or_get_image_file contains
* pretty general code, so could be moved from bf_usd project. */
#ifdef WITH_HYDRA
image_path = io::hydra::cache_or_get_image_file(bmain, scene, image, &tex->iuser);
Brecht Van Lommel commented 2023-09-07 18:38:17 +02:00
Owner
Copy Link

Can thiscallback function somehow that can be provided by the code that calls the MaterialX conversion?

That would also avoid the dependency of the shader nodes module on the USD module.

Can thiscallback function somehow that can be provided by the code that calls the MaterialX conversion? That would also avoid the dependency of the shader nodes module on the USD module.
BogdanNagirniak marked this conversation as resolved
Bogdan Nagirniak added 1 commit 2023-09-08 19:35:29 +02:00
Brecht Van Lommel commented 2023-09-11 12:11:14 +02:00
Owner
Copy Link

A more concrete proposal for how to move the code into the node files. There could be macros like these:

#define NODE_SHADER_MATERIALX_BEGIN \
class NodeMaterialXParser : materialx::NodeParser{ \
  NodeItem compute()

#define NODE_SHADER_MATERIALX_END \
};
\
void node_shader_materialx(bNode *node, NodeItem& result) \
{ \
  NodeMaterialXParser parser(bnode); \
  result = parser.compute(); \
}

Used like this in the node files:

namespace blender::nodes::node_shader_gamma_cc {

static void node_declare(NodeDeclarationBuilder &b)
{
  ..
}

NODE_SHADER_MATERIALX_BEGIN
{
  .. implementation ..
}
NODE_SHADER_MATERIALX_END

}  // namespace blender::nodes::node_shader_gamma_cc

void register_node_type_sh_gamma()
{
  namespace file_ns = blender::nodes::node_shader_gamma_cc;
  static bNodeType ntype;
  ..
  ntype.materialx = file_ns::node_shader_materialx
}
A more concrete proposal for how to move the code into the node files. There could be macros like these: ``` #define NODE_SHADER_MATERIALX_BEGIN \ class NodeMaterialXParser : materialx::NodeParser{ \ NodeItem compute() #define NODE_SHADER_MATERIALX_END \ }; \ void node_shader_materialx(bNode *node, NodeItem& result) \ { \ NodeMaterialXParser parser(bnode); \ result = parser.compute(); \ } ``` Used like this in the node files: ``` namespace blender::nodes::node_shader_gamma_cc { static void node_declare(NodeDeclarationBuilder &b) { .. } NODE_SHADER_MATERIALX_BEGIN { .. implementation .. } NODE_SHADER_MATERIALX_END } // namespace blender::nodes::node_shader_gamma_cc void register_node_type_sh_gamma() { namespace file_ns = blender::nodes::node_shader_gamma_cc; static bNodeType ntype; .. ntype.materialx = file_ns::node_shader_materialx } ```
Bogdan Nagirniak added 1 commit 2023-09-12 13:55:33 +02:00
9e483eace3 Move the MaterialX export code into the existing shader node files 
### Purpose
Due to review suggestions, MaterialX export code should be moved to existing node files.

### Technical steps
1. Added `bNodeType::materialx_fn` function pointer.
2. Created special macroses `NODE_SHADER_MATERIALX_BEGIN`, `NODE_SHADER_MATERIALX_END` for including MaterialX export code into `node_shader_<name>.cc` files.
3. Removed `ShaderNodeParser` as not needed, just `NodeParser` can be used. Adjusted `get_input_link()` to work with shader types. Made `get_input_link()` to call `materialx_fn`.
4. Moved `node_parser.cc/.h`, `node_item.cc/.h` to `materialx` folder.
5. Moved MaterialX node export implementation to corresponded `node_shader_<name>.cc` files. Removed `materialx/nodes/...` files.
6. Created `DefaultMaterialNodeParser` in `material.cc`.

Pull Request: DagerD/blender#18
Brecht Van Lommel requested changes 2023-09-12 17:48:40 +02:00
Brecht Van Lommel left a comment
Owner
Copy Link

Thanks, that reorganization looks good. Marking as request changes since there remain other comments to address.

Thanks, that reorganization looks good. Marking as request changes since there remain other comments to address.
source/blender/blenkernel/BKE_node.h Outdated
@ -225,6 +225,9 @@ typedef int (*NodeGPUExecFunction)(struct GPUMaterial *mat,
struct bNodeExecData *execdata,
struct GPUNodeStack *in,
struct GPUNodeStack *out);
typedef void (*NodeMaterialXExecFunction)(void *data,
Brecht Van Lommel commented 2023-09-12 14:18:11 +02:00
Owner
Copy Link

I would name this just NodeMaterialXFunction, the Exec is from a time when these functions actually executed the nodes.

I would name this just `NodeMaterialXFunction`, the `Exec` is from a time when these functions actually executed the nodes.
BogdanNagirniak marked this conversation as resolved
Bogdan Nagirniak added 2 commits 2023-09-14 04:29:53 +02:00
Bogdan Nagirniak added 1 commit 2023-09-14 16:55:20 +02:00
cbc1e652ab MaterialX: add shader nodes 
### Purpose
Add shader nodes

### Technical steps
Added nodes:
* TranslucentBSDF
* Subsurface scattering
* SheenBSDF

Pull Request: DagerD/blender#21
Bogdan Nagirniak added 1 commit 2023-09-15 20:11:06 +02:00
f6e00155f9 MaterialX: Implement more shader nodes 
### Purpose
Implement more shader nodes

### Technical steps
Added type: `NodeItem::Type::multioutput`
Added nodes:
* Glossy BSDF
* Glass BSDF
* Refraction BSDF

Pull Request: DagerD/blender#23
Bogdan Nagirniak added 1 commit 2023-09-18 12:49:25 +02:00
412b40266f Support group nodes 
### Purpose
Group nodes should be supported.

### Technical steps
1. Created classes `GroupNodeParser`, `GroupOutputNodeParser`, `GroupInputNodeParser` in `materialx/group_nodes.cc/.h`.
2. Added support of `NodeItem::input`, `NodeItem::output`.
3. Added control to use `<nodegraph>` by `#define USE_MATERIALX_NODEGRAPH`. Disabled by default because `pxr::UsdMtlxRead()` doesn't perform nodegraphs.
4. Added logging of `stage->ExportToString()` after `pxr::UsdMtlxRead()`.
5. Improvement in `node_shader_tex_noise.cc`.

Pull Request: DagerD/blender#22
Brecht Van Lommel commented 2023-09-18 13:09:40 +02:00
Owner
Copy Link

If you haven't already, please run the Hydra regression tests to identify problems. It will output a webpage where you can compare the results to cycles to see how closely it matches.

If you haven't already, please run the Hydra regression tests to identify problems. It will output a webpage where you can compare the results to cycles to see how closely it matches.
Bogdan Nagirniak added 1 commit 2023-09-19 18:28:09 +02:00
buildbot/vexp-code-patch-coordinator Build done. Details
2f980a9856 matx-extend-create_node 
### Purpose
Add parameter `inputs` to `NodeItem::create_node()` and simplify node creation in export implementation.

### Technical steps
Added type `NodeItem::Inputs`
Added parameter `inputs` to `NodeItem::create_node()`.
Updated node export implementation with new functionality.

Pull Request: DagerD/blender#25
Brecht Van Lommel commented 2023-09-19 19:42:43 +02:00
Owner
Copy Link

@blender-bot build

@blender-bot build
Bogdan Nagirniak commented 2023-09-20 12:14:44 +02:00
Author
Contributor
Copy Link

If you haven't already, please run the Hydra regression tests to identify problems. It will output a webpage where you can compare the results to cycles to see how closely it matches.

@brecht can you remind please how to do this? Which commands to run?

> If you haven't already, please run the Hydra regression tests to identify problems. It will output a webpage where you can compare the results to cycles to see how closely it matches. @brecht can you remind please how to do this? Which commands to run?
Brecht Van Lommel commented 2023-09-20 15:32:06 +02:00
Owner
Copy Link

https://wiki.blender.org/wiki/Tools/Tests/Setup

Assuming you have the test files downloaded.

  • Set WITH_OPENGL_RENDER_TESTS=ON and rebuild.
  • make test will work, but ctest -C Release -R storm from the build folder will be faster to run just the Storm tests
  • There will be a tests/report.html in the build folder with the results
https://wiki.blender.org/wiki/Tools/Tests/Setup Assuming you have the test files downloaded. * Set `WITH_OPENGL_RENDER_TESTS=ON` and rebuild. * `make test` will work, but `ctest -C Release -R storm` from the build folder will be faster to run just the Storm tests * There will be a `tests/report.html` in the build folder with the results
Brecht Van Lommel requested changes 2023-09-20 20:18:23 +02:00
Brecht Van Lommel left a comment
Owner
Copy Link

Please see the Linux build log for some warnings/errors to fix:
https://builder.blender.org/admin/#/builders/132/builds/2398

Group nodes don't seem to work reliably for me, two cases that didn't work:

  • In the default startup .blend, putting the Principled BSDF node in a group doesn't seem to work here. The cube renders white instead of the specified base color.
  • The shader/node_group_color.blend renders white when it should be green, ignoring the color set on the group.

There's some tricky logic with default values and socket type conversions that is not easy to get right and difficult to verify. That's why I suggested to simply expand the node groups to begin with. But it's up to you if really want to get this working well for 4.0.

Please see the Linux build log for some warnings/errors to fix: https://builder.blender.org/admin/#/builders/132/builds/2398 Group nodes don't seem to work reliably for me, two cases that didn't work: * In the default startup .blend, putting the Principled BSDF node in a group doesn't seem to work here. The cube renders white instead of the specified base color. * The `shader/node_group_color.blend` renders white when it should be green, ignoring the color set on the group. There's some tricky logic with default values and socket type conversions that is not easy to get right and difficult to verify. That's why I suggested to simply expand the node groups to begin with. But it's up to you if really want to get this working well for 4.0.
source/blender/io/usd/hydra/material.cc
@ -74,0 +82,4 @@
scene_delegate_->depsgraph, (Material *)id);
pxr::UsdMtlxRead(doc, stage);
/* Logging stage: creating lambda stage_str() for not to call stage->ExportToString()
Brecht Van Lommel commented 2023-09-20 20:06:17 +02:00
Owner
Copy Link

for not to -> to not

for not to -> to not
BogdanNagirniak marked this conversation as resolved
source/blender/nodes/shader/materialx/group_nodes.cc Outdated
@ -0,0 +68,4 @@
Vector<NodeItem> outputs;
for (int i = 0; i < values.size(); ++i) {
if (values[i]) {
outputs.append(create_output("output" + std::to_string(i + 1), values[i]));
Brecht Van Lommel commented 2023-09-20 20:12:22 +02:00
Owner
Copy Link

Can you explain why this is using output + index rather than output socket names?

I guess it's not to avoid naming conflicts, because output1 isn't unique enough that a user would never name their socket like this.

Can you explain why this is using `output + index` rather than output socket names? I guess it's not to avoid naming conflicts, because `output1` isn't unique enough that a user would never name their socket like this.
Bogdan Nagirniak commented 2023-09-24 12:55:09 +02:00
Author
Contributor
Copy Link

Changed name to out_<out socket name>

Changed name to `out_<out socket name>`
BogdanNagirniak marked this conversation as resolved
Bogdan Nagirniak added 1 commit 2023-09-21 09:31:35 +02:00
886b414a7c MaterialX: Implement Gradient Texture node. 
### Purpose
Add support for Gradient Texture node.

### Technical steps
Implemented Gradient Texture node according to existing Blender's implementation,
Fixed dotproduct, added `to_type` argument to be able to specify output type of node while using `arithmetic` method.

Pull Request: DagerD/blender#28
Brecht Van Lommel commented 2023-09-21 19:49:16 +02:00
Owner
Copy Link

A reminder that this should be merged before next Wednesday to get included in the 4.0 release.

At the moment it's unclear to me what exactly the status is, how many more changes are still planned. As long as this is marked "WIP" I assume this is not ready to merge yet.

A reminder that this should be merged before next Wednesday to get included in the 4.0 release. At the moment it's unclear to me what exactly the status is, how many more changes are still planned. As long as this is marked "WIP" I assume this is not ready to merge yet.
Bogdan Nagirniak commented 2023-09-21 23:04:44 +02:00
Author
Contributor
Copy Link

Thank you for reminder. We are almost on finish line.
Our status: implemented almost all nodes available in Cycles. Currently polishing, reviewing and testing their implementation. Trying to implement TODOs in code as much as possible.

TODOs:

  • check linux and MacOS builds
  • fix review comments
  • fix usage of io::hydra::cache_or_get_image_file
  • finish implementation of the remaining nodes: PRs are ready, will be merged tomorrow.
  • finish polishing of implemented nodes.

I think on Monday those TODOs will be ready except polishing and we'll be ready for merge.

Thank you for reminder. We are almost on finish line. Our status: implemented almost all nodes available in Cycles. Currently polishing, reviewing and testing their implementation. Trying to implement TODOs in code as much as possible. TODOs: - check linux and MacOS builds - fix review comments - fix usage of `io::hydra::cache_or_get_image_file` - finish implementation of the remaining nodes: PRs are ready, will be merged tomorrow. - finish polishing of implemented nodes. I think on Monday those TODOs will be ready except polishing and we'll be ready for merge.
Bogdan Nagirniak added 1 commit 2023-09-22 10:03:50 +02:00
74afa9c393 MaterialX: add support for Vector nodes 
### Purpose
Add support for Vector nodes.

### Technical steps
Fixed normal map color input type to Vector3.
Implemented NodeItem method rotate3d.
Implemented nodes

Added:
* Bump
* Mapping
* Normal
* Vector Rotate
* Vector Transform

Partially implemented (only node implementation):
* Vector Displacement
* Displacement

Not implemented:
* Vector Curves

Pull Request: DagerD/blender#27
Bogdan Nagirniak added 1 commit 2023-09-22 18:12:58 +02:00
d1492f8592 MaterialX: make caching image in MaterialX independent from `bf_usd` project 
### Purpose
Make caching image in MaterialX independent from `bf_usd` project

### Technical steps
* Added callback `ExportImageFunction` for image caching
* Removed `WITH_HYDRA` from CMakeLists

Pull Request: DagerD/blender#31
Bogdan Nagirniak added 1 commit 2023-09-22 18:23:17 +02:00
483728b704 Code improvements + Mix node 
### Purpose
After last merged PRs some code has to be improved. Implement Mix node.

### Technical steps
1. Improvements in NodeItem:
    - `blend()` -> `mix()` + shaders + mix node usage
    - `extract()` -> `operator[]`
    - implemented `length()`, `to_vector()`, `normalize()`, `rotate()`, fixed `dotproduct()`
    - improvements in math functions.
2. Implemented node Mix
3. Improved code in BSDFPrincipled, TexWave, TexGradient, Mapping, VectorRotate.
4. Code adjustments.
5. Fixed/added comments.
6. Changed order of getting shader type in MaterialOutput: SurfaceShader starts first. Adjusted Add, Mix shaders.

Pull Request: DagerD/blender#30
Bogdan Nagirniak added 1 commit 2023-09-24 07:42:40 +02:00
1132957f48 Fix linux build 
### Purpose
Errors in linux build https://builder.blender.org/admin/#/builders/132/builds/2398. Issues in group nodes.

### Technical steps
1. Fixed linux build: set math functions names to convenience.
2. Fixed some issues in group nodes. Made better naming of inputs and outputs in `nodegraph` for group node.
3. `make format`

Co-authored-by: Vasyl-Pidhirskyi <vpidhirskyi@gmail.com>
Pull Request: DagerD/blender#33
Bogdan Nagirniak added 3 commits 2023-09-24 12:52:25 +02:00
buildbot/vexp-code-patch-coordinator Build done. Details
d10771b96c Code adjustments after merge
Bogdan Nagirniak commented 2023-09-24 13:10:18 +02:00
Author
Contributor
Copy Link

TODOs:

  • check linux and MacOS builds
  • fix review comments
  • fix usage of io::hydra::cache_or_get_image_file
  • finish implementation of the remaining nodes: PRs are ready, will be merged tomorrow.
  • finish polishing of implemented nodes.

Most of issues were fixed. Polishing of implemented nodes are required + implement some remaining nodes. But this could be done after 4.0 release. Next several days we'll be spending on testing and bug fixing.

I think we are ready for merge at this stage. Please review.

> TODOs: > - check linux and MacOS builds > - fix review comments > - fix usage of `io::hydra::cache_or_get_image_file` > - finish implementation of the remaining nodes: PRs are ready, will be merged tomorrow. > - finish polishing of implemented nodes. Most of issues were fixed. Polishing of implemented nodes are required + implement some remaining nodes. But this could be done after 4.0 release. Next several days we'll be spending on testing and bug fixing. I think we are ready for merge at this stage. Please review.
Brecht Van Lommel commented 2023-09-24 13:54:40 +02:00
Owner
Copy Link

@blender-bot build

@blender-bot build
Brian Savery (AMD) commented 2023-09-25 02:05:09 +02:00
Member
Copy Link

TODOs:

  • check linux and MacOS builds
  • fix review comments
  • fix usage of io::hydra::cache_or_get_image_file
  • finish implementation of the remaining nodes: PRs are ready, will be merged tomorrow.
  • finish polishing of implemented nodes.

Most of issues were fixed. Polishing of implemented nodes are required + implement some remaining nodes. But this could be done after 4.0 release. Next several days we'll be spending on testing and bug fixing.

I think we are ready for merge at this stage. Please review.

Please document and not all nodes that are not supported yet.

> > TODOs: > > - check linux and MacOS builds > > - fix review comments > > - fix usage of `io::hydra::cache_or_get_image_file` > > - finish implementation of the remaining nodes: PRs are ready, will be merged tomorrow. > > - finish polishing of implemented nodes. > > Most of issues were fixed. Polishing of implemented nodes are required + implement some remaining nodes. But this could be done after 4.0 release. Next several days we'll be spending on testing and bug fixing. > > I think we are ready for merge at this stage. Please review. > > > > Please document and not all nodes that are not supported yet.
Ashley commented 2023-09-25 04:47:26 +02:00
Contributor
Copy Link

I've been testing this and it's pretty awesome! There are a few things that aren't at all blockers for getting this merged, but that I want to note for future work.

  • The 'closest' filtertype isn't respected at all. This is more of a MaterialX/Hydra issue: https://github.com/AcademySoftwareFoundation/MaterialX/issues/1525.
  • Multiple UV maps don't currently work, at the moment it just selects the active one.
  • There isn't any option to export MaterialX shaders yet, but it should be easy to add. They are printed to the terminal with the --log 'materialx.shader' flag.
  • When I tried to open a shader that was printed out, I got this error about subsurface_radius: Node interface error: Input 'subsurface_radius' doesn't match declaration: <standard_surface name="Principled_BSDF" type="surfaceshader">. This is probably due to a version mismatch and should be an easy fix.
I've been testing this and it's pretty awesome! There are a few things that aren't at all blockers for getting this merged, but that I want to note for future work. - The 'closest' filtertype isn't respected at all. This is more of a MaterialX/Hydra issue: https://github.com/AcademySoftwareFoundation/MaterialX/issues/1525. - Multiple UV maps don't currently work, at the moment it just selects the active one. - There isn't any option to export MaterialX shaders yet, but it should be easy to add. They are printed to the terminal with the `--log 'materialx.shader'` flag. - When I tried to open a shader that was printed out, I got this error about `subsurface_radius`: `Node interface error: Input 'subsurface_radius' doesn't match declaration: <standard_surface name="Principled_BSDF" type="surfaceshader">`. This is probably due to a version mismatch and should be an easy fix.
👍 1
Bogdan Nagirniak commented 2023-09-25 15:23:46 +02:00
Author
Contributor
Copy Link

Hello, we created document with statuses for all nodes available in Cycles
https://docs.google.com/spreadsheets/d/1U-jJ6QVpR4Dam6QDPoyUBYlScWQHYxfLGdDeX1eeNN8/edit#gid=0

Hello, we created document with statuses for all nodes available in Cycles https://docs.google.com/spreadsheets/d/1U-jJ6QVpR4Dam6QDPoyUBYlScWQHYxfLGdDeX1eeNN8/edit#gid=0
👍 1
Brecht Van Lommel added 1 commit 2023-09-25 18:31:25 +02:00
Brecht Van Lommel changed title from WIP: Export material to MaterialX for Hydra render to Export material to MaterialX for Hydra render 2023-09-25 18:32:42 +02:00
Brecht Van Lommel added 2 commits 2023-09-25 20:37:53 +02:00
buildbot/vexp-code-patch-coordinator Build done. Details
c2b5dfc2fe Update for renamed Principled BSDF sockets
Brian Savery (AMD) approved these changes 2023-09-25 20:42:44 +02:00
Brecht Van Lommel approved these changes 2023-09-25 20:50:50 +02:00
Brecht Van Lommel commented 2023-09-25 20:50:59 +02:00
Owner
Copy Link

@blender-bot build

@blender-bot build
Bogdan Nagirniak commented 2023-09-26 07:36:35 +02:00
Author
Contributor
Copy Link

When I tried to open a shader that was printed out, I got this error about subsurface_radius: Node interface error: Input 'subsurface_radius' doesn't match declaration: <standard_surface name="Principled_BSDF" type="surfaceshader">. This is probably due to a version mismatch and should be an easy fix.

This is fixed in DagerD/blender#34

> When I tried to open a shader that was printed out, I got this error about subsurface_radius: Node interface error: Input 'subsurface_radius' doesn't match declaration: <standard_surface name="Principled_BSDF" type="surfaceshader">. This is probably due to a version mismatch and should be an easy fix. This is fixed in https://projects.blender.org/DagerD/blender/pulls/34
Bogdan Nagirniak commented 2023-09-26 07:38:54 +02:00
Author
Contributor
Copy Link

There will be some PRs with node polishing today

There will be some PRs with node polishing today
Bogdan Nagirniak added 1 commit 2023-09-26 16:16:35 +02:00
6e1775d516 Implement BSDF Transparent shader 
### Purpose
Implement BSDF Transparent + include and shader's opacity to MatX nodetree.

### Technical steps
1. Added special type `NodeItem::Type::SurfaceOpacity` to retrieve opacity for \<surface> node.
2. Implemented export of BSDFTransparent node.
3. Implemented export of opacity component in BSDFPrincipled, Add/Mix shader nodes.
4. Removed redundant code.

### Notes
Storm and RPR hydra delegates aren't working good with opacity, this has to be fixed on delegate/hydra side, therefore those changes won't be visible. Opacity can be checked in MaterialXView or MaterialXGraphEditor of the exported mtlx.

Pull Request: DagerD/blender#36
Bogdan Nagirniak added 1 commit 2023-09-26 16:54:55 +02:00
buildbot/vexp-code-patch-coordinator Build done. Details
29944bc2eb Merge branch 'main' into matx-export-material
Bogdan Nagirniak commented 2023-09-26 17:23:20 +02:00
Author
Contributor
Copy Link

We've finished polishing code. Ready for merge

We've finished polishing code. Ready for merge
Brecht Van Lommel commented 2023-09-26 17:41:53 +02:00
Owner
Copy Link

@blender-bot build

@blender-bot build
Brecht Van Lommel commented 2023-09-26 17:46:55 +02:00
Owner
Copy Link

I copied the spreadsheet to #112864, since we like to track development on our own platform, and avoid referencing third party websites from release notes or the user manual.

I copied the spreadsheet to #112864, since we like to track development on our own platform, and avoid referencing third party websites from release notes or the user manual.
👍 1
Brecht Van Lommel manually merged commit c0a0de617c into main 2023-09-26 18:56:40 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Vasyl Pidhirskyi
Georgiy Markelov
Brian Savery (AMD)
Brecht Van Lommel
Labels
Clear labels   
Interest
Alembic

  
Interest
Animation & Rigging

  
Interest
Asset Browser

  
Interest
Asset Browser Project Overview

  
Interest
Audio

  
Interest
Automated Testing

  
Interest
Blender Asset Bundle

  
Interest
BlendFile

  
Interest
Collada

  
Interest
Compatibility
This issue affects/is about backward or forward compatibility

  
Interest
Compositing

  
Interest
Core

  
Interest
Cycles

  
Interest
Dependency Graph

  
Interest
Development Management

  
Interest
EEVEE

  
Interest
EEVEE & Viewport

  
Interest
Freestyle

  
Interest
Geometry Nodes

  
Interest
Grease Pencil

  
Interest
ID Management

  
Interest
Images & Movies

  
Interest
Import Export

  
Interest
Line Art

  
Interest
Masking

  
Interest
Metal

  
Interest
Modeling

  
Interest
Modifiers

  
Interest
Motion Tracking

  
Interest
Nodes & Physics

  
Interest
OpenGL

  
Interest
Overlay

  
Interest
Overrides

  
Interest
Performance

  
Interest
Physics

  
Interest
Pipeline, Assets & IO

  
Interest
Platforms, Builds & Tests

  
Interest
Python API

  
Interest
Render & Cycles

  
Interest
Render Pipeline

  
Interest
Sculpt, Paint & Texture

  
Interest
Text Editor

  
Interest
Translations

  
Interest
Triaging

  
Interest
Undo

  
Interest
USD

  
Interest
User Interface

  
Interest
UV Editing

  
Interest
VFX & Video

  
Interest
Video Sequencer

  
Interest
Virtual Reality

  
Interest
Vulkan

  
Interest
Wayland
Issues relating to Wayland windowing support.

  
Interest
Workbench

  
Interest: X11

Issues relating to Xorg/X11 support.

  
Legacy
Blender 2.8 Project

  
Legacy
Milestone 1: Basic, Local Asset Browser

  
Legacy
OpenGL Error

  
Meta
Good First Issue

  
Meta
Papercut

  
Meta
Retrospective

  
Meta
Security
Issues relating to security: https://wiki.blender.org/wiki/Process/Vulnerability_Reports

  
Module
Animation & Rigging

  
Module
Core

  
Module
Development Management

  
Module
EEVEE & Viewport

  
Module
Grease Pencil

  
Module
Modeling

  
Module
Nodes & Physics

  
Module
Pipeline, Assets & IO

  
Module
Platforms, Builds & Tests

  
Module
Python API

  
Module
Render & Cycles

  
Module
Sculpt, Paint & Texture

  
Module
Triaging

  
Module
User Interface

  
Module
VFX & Video

  
Platform
FreeBSD

  
Platform
Linux

  
Platform
macOS

  
Platform
Windows

  
Priority
High

  
Priority
Low

  
Priority
Normal

  
Priority
Unbreak Now!

  
Status
Archived

  
Status
Confirmed

  
Status
Duplicate

  
Status
Needs Info from Developers

  
Status
Needs Information from User

  
Status
Needs Triage

  
Status
Resolved

  
Type
Bug

  
Type
Design

  
Type
Known Issue

  
Type
Patch

  
Type
Report

  
Type
To Do

No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#111765
No description provided.
Delete Branch "DagerD/blender:matx-export-material"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?