Single frame job_compiler #104188

Open
opened 2023-02-21 22:42:46 +01:00 by k8ie · 3 comments

Hi, I'd like to ask if you'd consider adding a job_compiler for distributing single-frame renders.

To me this sounds like it is still a pretty general use case that many users could potentially find useful. I know of at least one project that has managed to successfully do this, BlendNet.

The job would (probably) work something like this:

  1. Job is submitted (the only parameters I can think of is output format, frame number and chunk size (number of samples in each task))
  2. Tasks are created with different sample ranges (with chunk size 256 the first task would be sample number 0 to 255, second 256-512, etc.)
  3. Task results are saved as OpenEXR multi-layer images
  4. Once rendering is finished, one of the nodes is tasked with stitching the image together using cycles.merge_images

I'm not sure how compositing or denoising would fit into this tho. I guess it should be skipped on the render tasks and only done on the final merged image.

Hi, I'd like to ask if you'd consider adding a job_compiler for distributing single-frame renders. To me this sounds like it is still a pretty general use case that many users could potentially find useful. I know of at least one project that has managed to successfully do this, [BlendNet](https://github.com/state-of-the-art/BlendNet). The job would (probably) work something like this: 1. Job is submitted (the only parameters I can think of is output format, frame number and chunk size (number of samples in each task)) 1. Tasks are created with different sample ranges (with chunk size 256 the first task would be sample number 0 to 255, second 256-512, etc.) 1. Task results are saved as OpenEXR multi-layer images 1. Once rendering is finished, one of the nodes is tasked with stitching the image together using [cycles.merge_images](https://docs.blender.org/api/current/bpy.ops.cycles.html#bpy.ops.cycles.merge_images) I'm not sure how compositing or denoising would fit into this tho. I guess it should be skipped on the render tasks and only done on the final merged image.

Thanks for your contribution! I don't have time to look into this right now, but do know that your PR is much appreciated. Hopefully I'll have some more time soon.

Thanks for your contribution! I don't have time to look into this right now, but do know that your PR is much appreciated. Hopefully I'll have some more time soon.
Sybren A. Stüvel added the
Priority
Normal
Status
Confirmed
Type
To Do
labels 2023-02-28 11:50:21 +01:00
Author

Hi, thank you for your response. No worries, I totally understand.

I'm still testing things out and ironing out some issues. Today I found that the chunking is not exactly correct. I'll mark remove the WIP when I think it's ready for review.

Hi, thank you for your response. No worries, I totally understand. I'm still testing things out and ironing out some issues. Today I found that the chunking is not exactly correct. I'll mark remove the WIP when I think it's ready for review.

I just had a talk with Sergey, and apparently with Cycles' adaptive sampling it's no longer feasible to do sample-based splitting. Tile-based splitting could work, if a large enough overscan is used (16-32 pixels should do it, but better to investigate properly before settling on a value). Each tile should be rendered to multilayer EXR, then merged, and then denoised. This work would have to be done in collaboration with Cycles developers, though, as not everything that's necessary for the denoising is actually written to multilayer EXR at the moment (for example some shadow catcher data).

I just had a talk with Sergey, and apparently with Cycles' adaptive sampling it's no longer feasible to do sample-based splitting. Tile-based splitting could work, if a large enough overscan is used (16-32 pixels should do it, but better to investigate properly before settling on a value). Each tile should be rendered to multilayer EXR, then merged, and then denoised. This work would have to be done in collaboration with Cycles developers, though, as not everything that's necessary for the denoising is actually written to multilayer EXR at the moment (for example some shadow catcher data).
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: studio/flamenco#104188
No description provided.