Distributed rendering of single images #104327

David Zhang · 2024-07-22T03:52:32+02:00

David Zhang commented

2024-07-22 03:52:32 +02:00

Overview

This pull request introduces a tile-based distributed rendering solution for single images, which would benefit artists working on complex scenes by reducing rendering times and improving image quality, without the need for extensive technical adjustments.

Steps

A new custom JOB_TYPE for single images
Rendering image in separate pieces in multilayer EXR, with overscan for adaptive sampling and with tile_size controllable by the users
Merging separate pieces into a whole image
Compositing on the image as a whole
Deal with denoising

**Overview** This pull request introduces a tile-based distributed rendering solution for single images, which would benefit artists working on complex scenes by reducing rendering times and improving image quality, without the need for extensive technical adjustments. **Steps** - A new custom `JOB_TYPE` for single images - Rendering image in separate pieces in multilayer EXR, with overscan for adaptive sampling and with `tile_size` controllable by the users - Merging separate pieces into a whole image - Compositing on the image as a whole - Deal with denoising

David Zhang added 1 commit 2024-07-22 03:52:37 +02:00

Manager: basic structure of the custom job type 187ca0d2b5

David Zhang added 1 commit 2024-07-22 05:35:13 +02:00

Manager: reformat and add render script ea6111e3ab

Sybren A. Stüvel reviewed 2024-07-23 09:39:23 +02:00

Sybren A. Stüvel left a comment

Low priority (highest one is to get things working ;-)) but I don't want to forget about this: the job type should record the current frame number, and explicitly render that frame.

When someone is working directly on the shared storage, there is no copy of the blend file for each render job. That means that they could be sending multiple jobs, rendering different frames from the same file. Using whatever was saved as the current frame then isn't enough to make that work.

Low priority (highest one is to get things working ;-)) but I don't want to forget about this: the job type should record the current frame number, and explicitly render that frame. When someone is working directly on the shared storage, there is no copy of the blend file for each render job. That means that they could be sending multiple jobs, rendering different frames from the same file. Using whatever was saved as the current frame then isn't enough to make that work.

David Zhang added 2 commits 2024-07-25 06:44:57 +02:00

Merge remote-tracking branch 'official/main' into single-image-render 0ddaff10c1

Manager: add frame field edcbe9f2fa

David Zhang added 1 commit 2024-07-26 05:28:27 +02:00

Manager: able to do border rendering ec0f9b47e2

David Zhang added 1 commit 2024-07-26 05:42:06 +02:00

Manager: add overscan a33bf01901

David Zhang added 1 commit 2024-07-28 22:54:25 +02:00

Manager: add merge task 3db9292e3b

David Zhang added 1 commit 2024-07-29 05:49:36 +02:00

Manager: add translate node cbbd481407

David Zhang added 1 commit 2024-08-05 04:14:03 +02:00

Manager: first working demo with some hacks 6fca87e42f

David Zhang added 2 commits 2024-08-10 21:14:41 +02:00

Merge remote-tracking branch 'official/main' into single-image-render 051210dc45

Manager: check for cases without Render Layers Node 8f63daf110

David Zhang added 1 commit 2024-08-10 21:53:37 +02:00

Manager: replace os with pathlib 7f79e74371

David Zhang added 1 commit 2024-08-26 03:55:11 +02:00

Manager: fix tile sizes of non-integer portion of resolution 6a51db78d7

David Zhang added 1 commit 2024-08-26 03:55:35 +02:00

Merge remote-tracking branch 'official/main' into single-image-render 049544cf49

Sybren A. Stüvel added this to the v3.6 milestone 2024-08-26 18:25:49 +02:00

David Zhang added 1 commit 2024-08-27 06:04:38 +02:00

Manager: minor changes 3dd46d1c00

David Zhang added 1 commit 2024-08-27 06:05:23 +02:00

Merge remote-tracking branch 'official/main' into single-image-render 51658e3847

David Zhang added 1 commit 2024-08-29 05:07:14 +02:00

Website: documentation for simple blender render 43b9f7a097

David Zhang added 1 commit 2024-08-29 06:19:09 +02:00

Website: documentation for single image render aeafe0fc6d

Sybren A. Stüvel commented

2024-08-29 12:52:03 +02:00

I've tried the job type, but for me it doesn't quite work well yet:

I've attached the file I used for rendering this: flamenco-bcon23-demo.blend

The job looks like:

{
  "metadata": {},
  "name": "cube-one-frame",
  "priority": 50,
  "settings": {
    "add_path_components": 0,
    "blendfile": "{jobs}/cube-one-frame-l7yf/flamenco-bcon23-demo.flamenco.blend",
    "format": "JPEG",
    "frame": 38,
    "image_file_extension": ".jpg",
    "render_output_path": "{shared_storage_mapping}/flamenco/renders/cube-one-frame/2024-08-29_124331/######",
    "render_output_root": "{shared_storage_mapping}/flamenco/renders/",
    "resolution_x": 2048,
    "resolution_y": 858,
    "tile_size_x": 512,
    "tile_size_y": 512
  },
  "storage": {
    "shaman_checkout_id": "cube-one-frame-l7yf"
  },
  "submitter_platform": "",
  "type": "single-image-render",
  "worker_tag": "ec60583f-f112-4e87-acab-79c79f3ba77f",
  "activity": "Changed to status \"completed\": all tasks completed",
  "created": "2024-08-29T10:43:31.11506402Z",
  "id": "d25ae08e-75ec-40c5-80a5-6a64818f4506",
  "status": "completed",
  "updated": "2024-08-29T10:43:54.294028483Z"
}

This is the output in the render directory:

.
├── ######
│   ├── 0-0-512-512.exr
│   ├── 0-512-512-858.exr
│   ├── 1024-0-1536-512.exr
│   ├── 1024-512-1536-858.exr
│   ├── 1536-0-2048-512.exr
│   ├── 1536-512-2048-858.exr
│   ├── 512-0-1024-512.exr
│   └── 512-512-1024-858.exr
└── merged.jpg

Those #-marks shouldn't be there, as those are typically an indicator that tells Blender where to put the frame number. Since Blender is not rendering an animation, that doesn't make sense to keep. Better to just replace it with a word like tiles.

The frame setting can be hidden in the submission interface, using visible: "web". And then moved down to the "Automatically evaluated settings".

I've tried the job type, but for me it doesn't quite work well yet: ![merged.jpg](/attachments/b9732e8b-ea84-4a11-b4e5-ba23203b4843) I've attached the file I used for rendering this: [flamenco-bcon23-demo.blend](/attachments/aedc4f21-80cc-4e99-b88a-4bcb55838658) The job looks like: ```json { "metadata": {}, "name": "cube-one-frame", "priority": 50, "settings": { "add_path_components": 0, "blendfile": "{jobs}/cube-one-frame-l7yf/flamenco-bcon23-demo.flamenco.blend", "format": "JPEG", "frame": 38, "image_file_extension": ".jpg", "render_output_path": "{shared_storage_mapping}/flamenco/renders/cube-one-frame/2024-08-29_124331/######", "render_output_root": "{shared_storage_mapping}/flamenco/renders/", "resolution_x": 2048, "resolution_y": 858, "tile_size_x": 512, "tile_size_y": 512 }, "storage": { "shaman_checkout_id": "cube-one-frame-l7yf" }, "submitter_platform": "", "type": "single-image-render", "worker_tag": "ec60583f-f112-4e87-acab-79c79f3ba77f", "activity": "Changed to status \"completed\": all tasks completed", "created": "2024-08-29T10:43:31.11506402Z", "id": "d25ae08e-75ec-40c5-80a5-6a64818f4506", "status": "completed", "updated": "2024-08-29T10:43:54.294028483Z" } ``` ---------- This is the output in the render directory: ``` . ├── ###### │ ├── 0-0-512-512.exr │ ├── 0-512-512-858.exr │ ├── 1024-0-1536-512.exr │ ├── 1024-512-1536-858.exr │ ├── 1536-0-2048-512.exr │ ├── 1536-512-2048-858.exr │ ├── 512-0-1024-512.exr │ └── 512-512-1024-858.exr └── merged.jpg ``` Those `#`-marks shouldn't be there, as those are typically an indicator that tells Blender where to put the frame number. Since Blender is not rendering an animation, that doesn't make sense to keep. Better to just replace it with a word like `tiles`. ------- The `frame` setting can be hidden in the submission interface, using `visible: "web"`. And then moved down to the "Automatically evaluated settings".

merged.jpg

35 KiB

flamenco-bcon23-demo.blend

522 KiB

David Zhang changed title from ~~WIP: Distributed rendering of single images~~ to Distributed rendering of single images

2024-08-30 03:02:14 +02:00

David Zhang added 1 commit 2024-08-30 05:26:23 +02:00

Manager: support resolution scale 1ee840bfb2

David Zhang added 1 commit 2024-08-30 05:30:10 +02:00

Manager: support stamp 0e08e81874

Sybren A. Stüvel commented

2024-08-30 11:36:15 +02:00

Now it's working, nice!

There does seem to be a math rounding issue somewhere though. This is what I get with:

the same test file as above
resolution scale = 100%
tile size 256 x 327 pixels

Now it's working, nice! There does seem to be a math rounding issue somewhere though. This is what I get with: - the same test file as above - resolution scale = 100% - tile size 256 x 327 pixels ![merged.jpg](/attachments/7ac68b5d-40b5-4159-9df7-602cc3488eb1)

merged.jpg

86 KiB

David Zhang added 1 commit 2024-09-02 13:02:55 +02:00

Manager: change path name 68a20578e7

David Zhang force-pushed single-image-render from 68a20578e7 to 2843993de4

2024-09-02 13:19:48 +02:00

Compare

David Zhang added 1 commit 2024-09-02 17:11:11 +02:00

Manager: explanatory comments 494ffe13a2

Sybren A. Stüvel requested review from Sybren A. Stüvel 2024-09-02 17:35:24 +02:00

Sybren A. Stüvel approved these changes 2024-09-02 17:41:26 +02:00

Sybren A. Stüvel left a comment

It seems to work! There's a bunch of improvements to make (like the division issue shown above), but let's land this before the GSoC is over, and look at those later.

David Zhang merged commit 60872c89f9 into main

2024-09-03 06:47:49 +02:00

David Zhang referenced this issue from a commit

2024-09-03 06:47:51 +02:00

Manager: distributed rendering of single images

Sign in to join this conversation.

No reviewers