Cycles/Blender API redesign #79131

New Issue

Kévin Dietrich · 2020-07-21T06:29:21+02:00

Kévin Dietrich commented

2020-07-21 06:29:21 +02:00

This design task is to propose and discuss ideas related to improvements to the Cycles API. Proposed changes will affect the Blender session and scene synchronization.

Problem

The current API to create a Cycles scene is too specific to the data types and exposes internal behaviors and class members to API users (clients hereafter). There exists a system in place, using the concept of Nodes and Sockets, to unify Cycles objects, and to generically manipulate them. The Nodes represent the objects (like a Mesh or a Light) and the Sockets represent their properties (like their vertices or intensity). However, it is not used throughout the API and clients have to directly manipulate certain properties in an unsafe manner. This direct manipulation has also two drawbacks related to scene updates. On one hand, we have to manually tag the objects or the system for an update; forgetting this can lead to missing objects or shaders in the render. On the other hand, there is no real way of telling what exactly has changed, so we may resort to making unnecessary work: simply modifying the vertex positions, without using displacement, should not free the device memory, but could simply copy the new positions in place (granularity of updates and fast scene rebuild shall be discussed in a separate task).

Ideally, the API should be simple, as generic as possible (i.e. to add the same shader to different nodes sharing a common property) while still allowing for strict typing (meaning we should not be able to set vertices on a camera if we know it is a Camera). Tagging for an update, and other bookkeeping behaviors if any, should become automatic. For this, we have to generalize the concept of Nodes and Sockets and use it throughout the public API. This will encapsulate internal behaviors for individual Nodes and the entire scene.

Abstraction

Every object accessible through the public API should become a Node, and public members of C++ classes in the current API, should be marked private.

Each node will have publicly accessible members exposed as sockets through setters/getters. Other member variables will be private and not be exposed as sockets.

Getters/Setters

Accessing sockets should be done via the various methods of the Node class. The different subclasses (Mesh, Curve, Light, etc.) could have more specific APIs to access and modify data.

For example, the vertices socket on the Mesh Node could be either accessed though Node::get_float3_array("vertices") or Mesh::get_vertices(). The specific APIs can get huge if done for every property, so it could either be constrained to only commonly used properties, or its generation could be automated somehow.

The reason to have Mesh::get_vertices() is for type safety, if the Cycles sockets change then there will be a compile error rather than a runtime error.

Updates

The X::tag_update(scene) will become automatic on socket (and attribute) changes. Each socket will have an associated bitflag, and when the socket value set function is called and the value is different, that bitflag will be set on the node.

Current logic in tag_update for indirectly tagging other nodes for update will be done later, as part of scene update.

Internally, some optimizations might be done to immediately also tag managers for update, or add updated nodes to a set so it is not required to loop over all nodes in scene update. This kind of optimization can be done as a second step.

D8644: Cycles: add update flags to Node and SocketType

Attributes

Geometry attributes are similar to sockets, but we prefer to keep them separate. This is to ensure there are no name collisions with sockets, and to support additional features and slightly different behavior where needed.

Attributes would use the same update flag mechanism. For performance reasons, comparing the old and new values detect if there was an actual changes would be optional. If the caller knows that the values has changed, it would be a waste of time to compare memory on a large array.

Further, we should support attributes taking ownership of array pointers rather than copying the array contents. For example for a system that already has vertex locations cached in memory for the entire animation, no expensive copy should be required.

Similarly for motion blur, we can have an efficient way to shift the motion blur steps, removing the first step and adding a new step at the end. For this each step would need to be stored in a separate array, rather than a single array with all time steps as is done now.

Creating Nodes and Ownership

Creating (and deleting) Nodes of various types should be done via an API (perhaps Scene::create_*_node()), instead of naively calling new and delete.

Nodes will optionally have an owner, to support #79174 (Cycles Procedural API and faster scene update). This ensures that when Alembic or USD procedurals create nodes, e.g. the Blender exporter will not try to delete them. Rather they would be owned by a procedural node, and deleted along with it.

D8540: Cycles: introduce an ownership system to define if nodes should be removed from the scene.

Node Types

The available node types would be the same as the existing ones, plus a few additional ones to make the whole system more consistent.

Image (move out of image manager and have a list of them in the scene, managed by the image manager) (D8649: Cycles: add an Image Node)
Volume (subclass of Geometry) (D8538: Cycles : add a Volume Geometry Node)
Pass/AOV (move out of Film and just have a list of them in the scene) (D8591: Cycles: add a Pass Node)
Session (D8751)
Device (actually just DeviceInfo) (D8750)
Procedural (new node that can dynamically generate nodes, see #79174)

Serialization

If the entire data structure is node based, serializing entire scenes becomes possible. However defining our own supported file format is not a goal, and we will likely just remove the XML support for now. Instead we should document the API well with examples.

Example

Here's an (incomplete) example on how rendering a simple scene would look like. The idea is to render something with the minimum amount of setup. This is using an approach where data is accessed using a specific API for each Node Type. A more complete example for the Mesh Node can be made.

int main()
{
    // data for a unit cube
    auto vertices_array = ccl::array<ccl::float3>({
            make_float3(-1.0f, -1.0f, -1.0f),
            make_float3( 1.0f, -1.0f, -1.0f),
            make_float3( 1.0f,  1.0f, -1.0f),
            make_float3(-1.0f,  1.0f, -1.0f),
            make_float3(-1.0f, -1.0f,  1.0f),
            make_float3( 1.0f, -1.0f,  1.0f),
            make_float3( 1.0f,  1.0f,  1.0f),
            make_float3(-1.0f,  1.0f,  1.0f)
    });

    auto triangles_array = ccl::array<ccl::int3>({ 
           make_int3(4, 0, 3),
           make_int3(4, 7, 3),
           make_int3(1, 5, 6),
           make_int3(1, 2, 6),
           make_int3(4, 5, 1),
           make_int3(4, 0, 1),
           make_int3(3, 2, 6),
           make_int3(3, 7, 6),
           make_int3(0, 1, 2),
           make_int3(0, 3, 2),
           make_int3(5, 4, 7),
           make_int3(5, 6, 7)
    });

    auto normals_array = ccl::array<ccl::float3>({ 
        make_float3(-1.0f,  0.0f,  0.0f),
        make_float3( 1.0f,  0.0f,  0.0f),
        make_float3( 0.0f, -1.0f,  0.0f),
        make_float3( 0.0f,  1.0f,  0.0f),
        make_float3( 0.0f,  0.0f, -1.0f),
        make_float3( 0.0f,  0.0f,  1.0f)
    });

    // set up the render session
    auto session = ccl::create_session()
    auto scene = session.get_scene();

    // create a mesh
    auto mesh = scene->create_mesh_node("my_mesh_name");
#if 0 // case for a generic API using the Node's methods
    mesh->set_float3_array("vertices", vertices_array);
    mesh->set_in3_array("triangles", triangles_array);
    mesh->set_float3_array("face_normals", normals_array);
#else // case for a specific Mesh API
    mesh->set_vertices(vertices_array);
    mesh->set_triangles(triangles_array);
    mesh->set_face_normals(normals_array);
#endif

    // create a shader
    auto shader = scene->create_shader_node("my_shader_name");
    *... setup the shader network, can have scene->get_default_diffuse_shader();, etc.* add the shader to the list of shaders used by the mesh
    // we could tell which triangles this shader affects, by default we assume the shader affects every triangles
    mesh->shaders().add(shader);

    auto camera = scene->create_camera_node("my_camera_name");
    *... setup the camera data* create device for the render
    auto device = scene->create_device_node();
    *.... setup the device data (requested feature, number of threads, etc.), can have multiple devices* do the render
    session.render();
}

This design task is to propose and discuss ideas related to improvements to the Cycles API. Proposed changes will affect the Blender session and scene synchronization. **Problem** The current API to create a Cycles scene is too specific to the data types and exposes internal behaviors and class members to API users (clients hereafter). There exists a system in place, using the concept of Nodes and Sockets, to unify Cycles objects, and to generically manipulate them. The Nodes represent the objects (like a Mesh or a Light) and the Sockets represent their properties (like their vertices or intensity). However, it is not used throughout the API and clients have to directly manipulate certain properties in an unsafe manner. This direct manipulation has also two drawbacks related to scene updates. On one hand, we have to manually tag the objects or the system for an update; forgetting this can lead to missing objects or shaders in the render. On the other hand, there is no real way of telling what exactly has changed, so we may resort to making unnecessary work: simply modifying the vertex positions, without using displacement, should not free the device memory, but could simply copy the new positions in place (granularity of updates and fast scene rebuild shall be discussed in a separate task). Ideally, the API should be simple, as generic as possible (i.e. to add the same shader to different nodes sharing a common property) while still allowing for strict typing (meaning we should not be able to set vertices on a camera if we know it is a Camera). Tagging for an update, and other bookkeeping behaviors if any, should become automatic. For this, we have to generalize the concept of Nodes and Sockets and use it throughout the public API. This will encapsulate internal behaviors for individual Nodes and the entire scene. **Abstraction** Every object accessible through the public API should become a Node, and public members of C++ classes in the current API, should be marked private. Each node will have publicly accessible members exposed as sockets through setters/getters. Other member variables will be private and not be exposed as sockets. **Getters/Setters** Accessing sockets should be done via the various methods of the Node class. The different subclasses (Mesh, Curve, Light, etc.) could have more specific APIs to access and modify data. For example, the `vertices` socket on the Mesh Node could be either accessed though `Node::get_float3_array("vertices")` or `Mesh::get_vertices()`. The specific APIs can get huge if done for every property, so it could either be constrained to only commonly used properties, or its generation could be automated somehow. The reason to have `Mesh::get_vertices()` is for type safety, if the Cycles sockets change then there will be a compile error rather than a runtime error. **Updates** The `X::tag_update(scene)` will become automatic on socket (and attribute) changes. Each socket will have an associated bitflag, and when the socket value set function is called and the value is different, that bitflag will be set on the node. Current logic in `tag_update` for indirectly tagging other nodes for update will be done later, as part of scene update. Internally, some optimizations might be done to immediately also tag managers for update, or add updated nodes to a set so it is not required to loop over all nodes in scene update. This kind of optimization can be done as a second step. [D8644: Cycles: add update flags to Node and SocketType](https://archive.blender.org/developer/D8644) **Attributes** Geometry attributes are similar to sockets, but we prefer to keep them separate. This is to ensure there are no name collisions with sockets, and to support additional features and slightly different behavior where needed. Attributes would use the same update flag mechanism. For performance reasons, comparing the old and new values detect if there was an actual changes would be optional. If the caller knows that the values has changed, it would be a waste of time to compare memory on a large array. Further, we should support attributes taking ownership of array pointers rather than copying the array contents. For example for a system that already has vertex locations cached in memory for the entire animation, no expensive copy should be required. Similarly for motion blur, we can have an efficient way to shift the motion blur steps, removing the first step and adding a new step at the end. For this each step would need to be stored in a separate array, rather than a single array with all time steps as is done now. **Creating Nodes and Ownership** Creating (and deleting) Nodes of various types should be done via an API (perhaps `Scene::create_*_node()`), instead of naively calling `new` and `delete`. Nodes will optionally have an owner, to support #79174 (Cycles Procedural API and faster scene update). This ensures that when Alembic or USD procedurals create nodes, e.g. the Blender exporter will not try to delete them. Rather they would be owned by a procedural node, and deleted along with it. [D8540: Cycles: introduce an ownership system to define if nodes should be removed from the scene.](https://archive.blender.org/developer/D8540) **Node Types** The available node types would be the same as the existing ones, plus a few additional ones to make the whole system more consistent. - [ ] Image (move out of image manager and have a list of them in the scene, managed by the image manager) ([D8649: Cycles: add an Image Node](https://archive.blender.org/developer/D8649)) - [x] Volume (subclass of Geometry) ([D8538: Cycles : add a Volume Geometry Node](https://archive.blender.org/developer/D8538)) - [x] Pass/AOV (move out of Film and just have a list of them in the scene) ([D8591: Cycles: add a Pass Node](https://archive.blender.org/developer/D8591)) - [ ] Session ([D8751](https://archive.blender.org/developer/D8751)) - [ ] Device (actually just DeviceInfo) ([D8750](https://archive.blender.org/developer/D8750)) - [ ] Procedural (new node that can dynamically generate nodes, see #79174) **Serialization** If the entire data structure is node based, serializing entire scenes becomes possible. However defining our own supported file format is not a goal, and we will likely just remove the XML support for now. Instead we should document the API well with examples. **Example** Here's an (incomplete) example on how rendering a simple scene would look like. The idea is to render something with the minimum amount of setup. This is using an approach where data is accessed using a specific API for each Node Type. A more complete example for the Mesh Node can be made. ``` int main() { // data for a unit cube auto vertices_array = ccl::array<ccl::float3>({ make_float3(-1.0f, -1.0f, -1.0f), make_float3( 1.0f, -1.0f, -1.0f), make_float3( 1.0f, 1.0f, -1.0f), make_float3(-1.0f, 1.0f, -1.0f), make_float3(-1.0f, -1.0f, 1.0f), make_float3( 1.0f, -1.0f, 1.0f), make_float3( 1.0f, 1.0f, 1.0f), make_float3(-1.0f, 1.0f, 1.0f) }); auto triangles_array = ccl::array<ccl::int3>({ make_int3(4, 0, 3), make_int3(4, 7, 3), make_int3(1, 5, 6), make_int3(1, 2, 6), make_int3(4, 5, 1), make_int3(4, 0, 1), make_int3(3, 2, 6), make_int3(3, 7, 6), make_int3(0, 1, 2), make_int3(0, 3, 2), make_int3(5, 4, 7), make_int3(5, 6, 7) }); auto normals_array = ccl::array<ccl::float3>({ make_float3(-1.0f, 0.0f, 0.0f), make_float3( 1.0f, 0.0f, 0.0f), make_float3( 0.0f, -1.0f, 0.0f), make_float3( 0.0f, 1.0f, 0.0f), make_float3( 0.0f, 0.0f, -1.0f), make_float3( 0.0f, 0.0f, 1.0f) }); // set up the render session auto session = ccl::create_session() auto scene = session.get_scene(); // create a mesh auto mesh = scene->create_mesh_node("my_mesh_name"); #if 0 // case for a generic API using the Node's methods mesh->set_float3_array("vertices", vertices_array); mesh->set_in3_array("triangles", triangles_array); mesh->set_float3_array("face_normals", normals_array); #else // case for a specific Mesh API mesh->set_vertices(vertices_array); mesh->set_triangles(triangles_array); mesh->set_face_normals(normals_array); #endif // create a shader auto shader = scene->create_shader_node("my_shader_name"); *... setup the shader network, can have scene->get_default_diffuse_shader();, etc.* add the shader to the list of shaders used by the mesh // we could tell which triangles this shader affects, by default we assume the shader affects every triangles mesh->shaders().add(shader); auto camera = scene->create_camera_node("my_camera_name"); *... setup the camera data* create device for the render auto device = scene->create_device_node(); *.... setup the device data (requested feature, number of threads, etc.), can have multiple devices* do the render session.render(); } ```

Kévin Dietrich commented

2020-07-21 06:29:21 +02:00

Added subscriber: @kevindietrich

Ray molenkamp commented

2020-07-21 16:11:02 +02:00

Added subscriber: @LazyDodo

Ray molenkamp commented

2020-07-21 16:11:02 +02:00

First of all I love the approach of starting from a hypothetical client using the API.

    auto triangles_array = ccl::array<ccl::float3>({ 
           make_int3(4, 0, 3),

This seems somewhat counter intuitive, perhaps ccl::array<ccl::int3> was meant here?

    auto device = scene->create_device_node();

This bit will need a bit of work, as the example is currently structured this would probably be more intuitive in session rather than scene , scene ideally is a scene description, it ought to not concern it self with hardware enumeration/configuration.

First of all I love the approach of starting from a hypothetical client using the API. ``` auto triangles_array = ccl::array<ccl::float3>({ make_int3(4, 0, 3), ``` This seems somewhat counter intuitive, perhaps `ccl::array<ccl::int3>` was meant here? ``` auto device = scene->create_device_node(); ``` This bit will need a bit of work, as the example is currently structured this would probably be more intuitive in `session` rather than `scene` , scene ideally is a scene description, it ought to not concern it self with hardware enumeration/configuration.

feng xie commented

2020-07-21 18:12:01 +02:00

Added subscriber: @fx

Kenzie commented

2020-07-22 06:12:42 +02:00

Added subscriber: @KenzieMac130

Kévin Dietrich commented

2020-07-22 22:03:12 +02:00

In #79131#982870, @LazyDodo wrote:
This seems somewhat counter intuitive, perhaps ccl::array<ccl::int3> was meant here?

Ah yes, it's a typo.

    auto device = scene->create_device_node();
This bit will need a bit of work, as the example is currently structured this would probably be more intuitive in session rather than scene , scene ideally is a scene description, it ought to not concern it self with hardware enumeration/configuration.

I agree, the idea is to make everything a node and to centralize a bit node creation, since most nodes are in the scene it made sense to have the scene create all of them. The Scene already contains a pointer to the Device, however it does not create it.

> In #79131#982870, @LazyDodo wrote: > This seems somewhat counter intuitive, perhaps `ccl::array<ccl::int3>` was meant here? Ah yes, it's a typo. > ``` > auto device = scene->create_device_node(); > ``` > This bit will need a bit of work, as the example is currently structured this would probably be more intuitive in `session` rather than `scene` , scene ideally is a scene description, it ought to not concern it self with hardware enumeration/configuration. I agree, the idea is to make everything a node and to centralize a bit node creation, since most nodes are in the scene it made sense to have the scene create all of them. The Scene already contains a pointer to the Device, however it does not create it.

blender-admin commented

2020-08-18 12:15:46 +02:00

This issue was referenced by blender/cycles@5c8fb3c37c

This issue was referenced by blender/cycles@5c8fb3c37c7438b7c625983b3eb702d53ccdf997

blender-admin commented

2020-08-19 15:46:50 +02:00

This issue was referenced by blender/cycles@4c672f0163

This issue was referenced by blender/cycles@4c672f016374f243b0185a0a85dc6b599e5b0cd5

blender-admin commented

2020-08-19 15:46:50 +02:00

This issue was referenced by aa1e4baa22

This issue was referenced by aa1e4baa22b3393dc723d48061c9781f4b8b42c7

blender-admin commented

2020-08-30 23:20:51 +02:00

This issue was referenced by blender/cycles@c19302e499

This issue was referenced by blender/cycles@c19302e499c2792b911cbad9374f96bd89897347

blender-admin commented

2020-08-30 23:20:51 +02:00

This issue was referenced by 429afe0c62

This issue was referenced by 429afe0c626a6d608385c6bc3a348b3ac8cfa8c0

blender-admin commented

2020-09-22 14:47:10 +02:00

This issue was referenced by blender/cycles@d3720f188c

This issue was referenced by blender/cycles@d3720f188c5231e330890755bd75f1199f3cafed

blender-admin commented

2020-09-22 14:47:10 +02:00

This issue was referenced by 626201683e

This issue was referenced by 626201683ec0d96e86d681bb0273a324cdf91916

Bartosz Moniewski commented

2020-09-29 21:41:56 +02:00

Added subscriber: @BartekMoniewski

Larry Weinberg commented

2020-10-15 07:14:31 +02:00

Added subscriber: @larrywberg

Larry Weinberg commented

2020-10-15 07:14:31 +02:00

Hi Guys. My first time here on the forum.

I would like to put in a vote to keep xml parsing compiling if it's not causing specific problems.

I am currently porting the latest cycles into a new version of Poser and have been fleshing out the xml for our use. I could share back to the community any parts of it that would be commonly useful if you keep the xml files in place.

Hi Guys. My first time here on the forum. I would like to put in a vote to keep xml parsing compiling if it's not causing specific problems. I am currently porting the latest cycles into a new version of Poser and have been fleshing out the xml for our use. I could share back to the community any parts of it that would be commonly useful if you keep the xml files in place.

Brecht Van Lommel commented

2020-10-28 14:57:28 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Brecht Van Lommel commented

2021-02-09 17:49:56 +01:00

Added subscriber: @brecht

Brecht Van Lommel commented

2021-02-09 17:49:56 +01:00

So @kevindietrich has done a bunch of work to get us closer to the code sample in the original post, but we're not there yet.

We still have the concept of SceneParams, SessionParams which are not exactly like nodes. The Session needs to be recreated when adding devices or enabled denoising. And it's not obvious when you can edit a Session's scene, how to mutex luck, when to reset, etc. There's still a bunch of tag_update calls that need to be made when changing certain data. And for denoising the user needs to manually sync some data between various Cycles data structures.

Ideally we can get towards a state where the scene and it's nodes can just be updated, and then the Session can figure out what to do in response to that. Restart the render if needed, recreate devices, etc.

Following the above example code, dynamically updating the scene in an interactive session should ideally be something like this.

*Acquire the scene for editing, pausing any render operations that might be accessing the scene data structures.* The scene pointer is only available here, and should not be stored permanently.
Scene& scene = session.begin_scene_edit();

*Make arbitrary edits to the scene. There is no tag_update() here.* Session responds to scene changes.
*- Flushing update tags between nodes* - (Re)creating devices
*- Scene device update* - Restarting the render from sample 0
** These may also be deferred to a running render thread if there is one, or until one is created.
session.end_scene_edit(scene);

I did some work towards this in a branch, and will post some WIP patches soon. Adding the basic begin/end scene edit mechanism, eliminating the need to call tag_update(), and did some work towards making denoising and passes be handled more automatically.

However there's still some significant roadblocks towards getting it this simple, and I'm not sure it's entirely practical. For example recreating devices (also for denoising) or changing the shading system between SVM/OSL seems impractical. That's because they persistently modify scene data and do not keep around the original for memory efficiency and ease of implementation. So I think there are a few high-level parameters that we may not allow changing dynamically, and that will continue to require recreating the session in an interactive session. But we should be able to minimize and document them.

So @kevindietrich has done a bunch of work to get us closer to the code sample in the original post, but we're not there yet. We still have the concept of `SceneParams`, `SessionParams` which are not exactly like nodes. The Session needs to be recreated when adding devices or enabled denoising. And it's not obvious when you can edit a Session's scene, how to mutex luck, when to reset, etc. There's still a bunch of `tag_update` calls that need to be made when changing certain data. And for denoising the user needs to manually sync some data between various Cycles data structures. Ideally we can get towards a state where the scene and it's nodes can just be updated, and then the Session can figure out what to do in response to that. Restart the render if needed, recreate devices, etc. Following the above example code, dynamically updating the scene in an interactive session should ideally be something like this. ```lang=c *Acquire the scene for editing, pausing any render operations that might be accessing the scene data structures.* The scene pointer is only available here, and should not be stored permanently. Scene& scene = session.begin_scene_edit(); *Make arbitrary edits to the scene. There is no tag_update() here.* Session responds to scene changes. *- Flushing update tags between nodes* - (Re)creating devices *- Scene device update* - Restarting the render from sample 0 ** These may also be deferred to a running render thread if there is one, or until one is created. session.end_scene_edit(scene); ``` I did some work towards this in a branch, and will post some WIP patches soon. Adding the basic begin/end scene edit mechanism, eliminating the need to call `tag_update()`, and did some work towards making denoising and passes be handled more automatically. However there's still some significant roadblocks towards getting it this simple, and I'm not sure it's entirely practical. For example recreating devices (also for denoising) or changing the shading system between SVM/OSL seems impractical. That's because they persistently modify scene data and do not keep around the original for memory efficiency and ease of implementation. So I think there are a few high-level parameters that we may not allow changing dynamically, and that will continue to require recreating the session in an interactive session. But we should be able to minimize and document them.

Brecht Van Lommel added this to the Render & Cycles project 2023-02-07 19:08:35 +01:00

Thomas Dinges added this to the 2.90 milestone 2023-02-08 16:25:00 +01:00

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Cycles/Blender API redesign #79131