Single frame job_compiler #104188

New Issue

k8ie · 2023-02-21T22:42:46+01:00

k8ie commented

2023-02-21 22:42:46 +01:00

Hi, I'd like to ask if you'd consider adding a job_compiler for distributing single-frame renders.

To me this sounds like it is still a pretty general use case that many users could potentially find useful. I know of at least one project that has managed to successfully do this, BlendNet.

The job would (probably) work something like this:

Job is submitted (the only parameters I can think of is output format, frame number and chunk size (number of samples in each task))
Tasks are created with different sample ranges (with chunk size 256 the first task would be sample number 0 to 255, second 256-512, etc.)
Task results are saved as OpenEXR multi-layer images
Once rendering is finished, one of the nodes is tasked with stitching the image together using cycles.merge_images

I'm not sure how compositing or denoising would fit into this tho. I guess it should be skipped on the render tasks and only done on the final merged image.

Hi, I'd like to ask if you'd consider adding a job_compiler for distributing single-frame renders. To me this sounds like it is still a pretty general use case that many users could potentially find useful. I know of at least one project that has managed to successfully do this, [BlendNet](https://github.com/state-of-the-art/BlendNet). The job would (probably) work something like this: 1. Job is submitted (the only parameters I can think of is output format, frame number and chunk size (number of samples in each task)) 1. Tasks are created with different sample ranges (with chunk size 256 the first task would be sample number 0 to 255, second 256-512, etc.) 1. Task results are saved as OpenEXR multi-layer images 1. Once rendering is finished, one of the nodes is tasked with stitching the image together using [cycles.merge_images](https://docs.blender.org/api/current/bpy.ops.cycles.html#bpy.ops.cycles.merge_images) I'm not sure how compositing or denoising would fit into this tho. I guess it should be skipped on the render tasks and only done on the final merged image.

k8ie referenced this issue

2023-02-27 21:16:06 +01:00

WIP: Initial version of a single-frame job compiler #104189

Sybren A. Stüvel commented

2023-02-28 11:50:08 +01:00

Thanks for your contribution! I don't have time to look into this right now, but do know that your PR is much appreciated. Hopefully I'll have some more time soon.

Sybren A. Stüvel added the

labels 2023-02-28 11:50:21 +01:00

k8ie commented

2023-02-28 12:29:04 +01:00

Hi, thank you for your response. No worries, I totally understand.

I'm still testing things out and ironing out some issues. Today I found that the chunking is not exactly correct. I'll mark remove the WIP when I think it's ready for review.

Hi, thank you for your response. No worries, I totally understand. I'm still testing things out and ironing out some issues. Today I found that the chunking is not exactly correct. I'll mark remove the WIP when I think it's ready for review.

k8ie referenced this issue

2023-03-06 14:50:03 +01:00

WIP: Single-frame job compiler #104194

Sybren A. Stüvel commented

2024-06-19 15:55:39 +02:00

I just had a talk with Sergey, and apparently with Cycles' adaptive sampling it's no longer feasible to do sample-based splitting. Tile-based splitting could work, if a large enough overscan is used (16-32 pixels should do it, but better to investigate properly before settling on a value). Each tile should be rendered to multilayer EXR, then merged, and then denoised. This work would have to be done in collaboration with Cycles developers, though, as not everything that's necessary for the denoising is actually written to multilayer EXR at the moment (for example some shadow catcher data).

k8ie referenced this issue

2024-08-20 19:19:42 +02:00

WIP: Single-frame job compiler #104194

Sign in to join this conversation.