WIP: Cycles: Importance Resampling #121023

Weizhen Huang · 2024-04-24T15:19:10+02:00

Weizhen Huang commented

2024-04-24 15:19:10 +02:00

Implement Reservoir-based Spatial and Temporal Importance Resampling, aka ReSTIR.
This is heavily WIP and inefficient, currently only implemented initial and spatial resampling for direct illumination, works on CPU with surface lit by area light, sphere light and mesh light.

Direct Light Only	Before (5 or 4 spp)	After (1 spp)
7.4 s, with light tree
~4.5s, no light tree

TODO:

Multiple spatial sampling iterations
Improve spatial sampling MIS weight to remove fireflies
Temporal resampling (ReSTIR)
Global Illumination (ReSTIR PT)
Shift mapping (different sampling method based on roughness and light types, also for direct illumination)
Respect MIS setting of light
Reduce reservoir pass storage
- Random replay from camera
- Compress data
Increase speed
- Move BSDF sampling outside of NEE
- Check visibility only when necessary
- Use light tiles to reduce cache thrash
Init from bake (?)
Properly handle tiled rendering
Handle noise threshold
Schedule multiple samples as before
Volume
Transparent surfaces (properly check visibility)
Subsurface
Curve and point cloud
Improve sphere light with soft falloff
Spot light
Background light
Distant light
Support GPU
Add back MNEE (or can we replace MNEE?)
Check compatibility with path guiding
Check light passes (diffuse/glossy)
Check the case when use_direct_light is false
Support ray portal
Find good default values of the setting (Different for viewport and render. Multiples spatial iterations seem to cause bias)
Verify if material with multiple closures are working correctly (what if different lobes and sampled? do we want to evaluate all closures during sampling?)

FIX:

Bias in spatial resampling due to
- wrong formulation of the visibility term
Denoiser not working with spatial resampling
Random crashes with NaN (?)
Pure specular surface not rendering (can not record ls and eval later)

Cleanup:

Unify measures in solid angle and area (the existing functions operate in solid angle measure, we need to convert it to area measure for spatial resampling. Should we separate the evaluation function for spatial resampling or just always use area measure to avoid confusion?)
Use buffer instead of render_buffer if it is already available
Add ReSTIR kernel feature

Debug options:

Implement Reservoir-based Spatial and Temporal Importance Resampling, aka **ReSTIR**. This is heavily WIP and inefficient, currently only implemented initial and spatial resampling for direct illumination, works on CPU with surface lit by area light, sphere light and mesh light. | Direct Light Only | Before (5 or 4 spp) | After (1 spp) | | -------- | -------- | -------- | | 7.4 s, with light tree | ![before_7.41s_5spp.png](/attachments/66f1562d-01f2-4bec-afbe-8f9a361f29e1) | ![after_7.49s_1spp.png](/attachments/5c14bd79-c589-4939-ac6f-4038d6de7e93) | |~4.5s, no light tree|![before_4.84s_4spp_no_tree.png](/attachments/482ddbed-6d2a-412b-8d3f-84cc9a690ea3)|![after_4.35s_1spp_no_tree.png](/attachments/b866029f-b39e-472a-abd3-1bb3bd1a773f)| **TODO:** - [x] Multiple spatial sampling iterations - [x] Improve spatial sampling MIS weight to remove fireflies - [ ] Temporal resampling ([ReSTIR](https://research.nvidia.com/sites/default/files/pubs/2020-07_Spatiotemporal-reservoir-resampling/ReSTIR.pdf)) - [ ] Global Illumination ([ReSTIR PT](https://research.nvidia.com/publication/2022-07_generalized-resampled-importance-sampling-foundations-restir)) - [ ] Shift mapping (different sampling method based on roughness and light types, also for direct illumination) - [ ] Respect MIS setting of light - [ ] Reduce reservoir pass storage - [x] Random replay from camera - [ ] Compress data - [ ] Increase speed - [x] Move BSDF sampling outside of NEE - [ ] Check visibility only when necessary - [ ] Use light tiles to reduce cache thrash - [ ] Init from bake (?) - [ ] Properly handle tiled rendering - [ ] Handle noise threshold - [ ] Schedule multiple samples as before - [ ] Volume - [ ] Transparent surfaces (properly check visibility) - [ ] Subsurface - [ ] Curve and point cloud - [ ] Improve sphere light with soft falloff - [ ] Spot light - [ ] Background light - [ ] Distant light - [ ] Support GPU - [ ] Add back MNEE (or can we replace MNEE?) - [ ] Check compatibility with path guiding - [ ] Check light passes (diffuse/glossy) - [ ] Check the case when `use_direct_light` is false - [ ] Support ray portal - [ ] Find good default values of the setting (Different for viewport and render. Multiples spatial iterations seem to cause bias) - [ ] Verify if material with multiple closures are working correctly (what if different lobes and sampled? do we want to evaluate all closures during sampling?) **FIX:** - [ ] Bias in spatial resampling due to - [ ] wrong formulation of the visibility term - [x] Denoiser not working with spatial resampling - [ ] Random crashes with NaN (?) - [ ] Pure specular surface not rendering (can not record ls and eval later) **Cleanup:** - [ ] Unify measures in solid angle and area (the existing functions operate in solid angle measure, we need to convert it to area measure for spatial resampling. Should we separate the evaluation function for spatial resampling or just always use area measure to avoid confusion?) - [ ] Use `buffer` instead of `render_buffer` if it is already available - [ ] Add ReSTIR kernel feature **Debug options:** ![Screenshot 2024-04-24 at 15.10.56.png](/attachments/53d175e8-54a7-4e4f-a990-f8f2ea07afd4)

before_4.84s_4spp_no_tree.png

3.4 MiB

before_7.41s_5spp.png

2.6 MiB

after_7.49s_1spp.png

4.4 MiB

after_4.35s_1spp_no_tree.png

2.5 MiB

Screenshot 2024-04-24 at 15.10.56.png

239 KiB

👍 14 👀 10 ❤️ 28 🎉 11 🚀 10

Weizhen Huang added this to the Render & Cycles project 2024-04-24 15:19:26 +02:00

Weizhen Huang force-pushed restir from 0b83af2c4d to 0023d17cb7

2024-04-26 16:10:14 +02:00

Compare

Weizhen Huang force-pushed restir from ec6bd6ca21 to cb15fa32dc

2024-04-30 18:31:41 +02:00

Compare

Weizhen Huang force-pushed restir from 3968b522ac to c3bc745a6b

2024-05-02 15:17:03 +02:00

Compare

Weizhen Huang force-pushed restir from 726e33588d to 1057d09633

2024-05-08 19:37:07 +02:00

Compare

Weizhen Huang force-pushed restir from 1057d09633 to 8a03cab90b

2024-05-13 19:27:03 +02:00

Compare

Weizhen Huang force-pushed restir from 2a4c28cd1f to 8b4acc8562

2024-05-16 22:21:31 +02:00

Compare

Weizhen Huang force-pushed restir from 8b1a78a879 to d1ebc8d202

2024-05-27 10:19:21 +02:00

Compare

JohnDow commented

2024-05-27 11:56:20 +02:00

First-time contributor

A suggestion to checkout the paper "Area ReSTIR: Resampling for Real-Time Defocus and Antialiasing"

@weizhen in case you've missed:

Area ReSTIR: Resampling for Real-Time Defocus and Antialiasing

Recent advancements in spatiotemporal reservoir resampling (ReSTIR) leverage sample reuse from neighbors to efficiently evaluate the path integral. Like rasterization, ReSTIR methods implicitly assume a pinhole camera and evaluate the light arriving at a pixel through a single predetermined subpixel location at a time (e.g., the pixel center). This prevents efficient path reuse in and near pixels with high-frequency details.

We introduce Area ReSTIR, extending ReSTIR reservoirs to also integrate each pixel's 4D ray space, including 2D areas on the film and lens. We design novel subpixel-tracking temporal reuse and shift mappings that maximize resampling quality in such regions. This robustifies ReSTIR against high-frequency content, letting us importance sample subpixel and lens coordinates and efficiently render antialiasing and depth of field.

And...

Source Code (Direct Lighting Part) for SIGGRAPH 2024 Paper "Area ReSTIR: Resampling for Real-Time Defocus and Antialiasing"

<details> <summary>A suggestion to checkout the paper "Area ReSTIR: Resampling for Real-Time Defocus and Antialiasing"</summary> @weizhen in case you've missed: ## [Area ReSTIR: Resampling for Real-Time Defocus and Antialiasing](https://graphics.cs.utah.edu/research/projects/area-restir/) ![](https://graphics.cs.utah.edu/research/projects/area-restir/restir-vs-arearestir-small.jpg) ``` Recent advancements in spatiotemporal reservoir resampling (ReSTIR) leverage sample reuse from neighbors to efficiently evaluate the path integral. Like rasterization, ReSTIR methods implicitly assume a pinhole camera and evaluate the light arriving at a pixel through a single predetermined subpixel location at a time (e.g., the pixel center). This prevents efficient path reuse in and near pixels with high-frequency details. We introduce Area ReSTIR, extending ReSTIR reservoirs to also integrate each pixel's 4D ray space, including 2D areas on the film and lens. We design novel subpixel-tracking temporal reuse and shift mappings that maximize resampling quality in such regions. This robustifies ReSTIR against high-frequency content, letting us importance sample subpixel and lens coordinates and efficiently render antialiasing and depth of field. ``` And... [Source Code (Direct Lighting Part) for SIGGRAPH 2024 Paper "Area ReSTIR: Resampling for Real-Time Defocus and Antialiasing"](https://github.com/guiqi134/Area-ReSTIR) </details>

👍 2 👀 6

Weizhen Huang force-pushed restir from d1ebc8d202 to 409d2ec1f1

2024-06-02 17:16:38 +02:00

Compare

Weizhen Huang force-pushed restir from 409d2ec1f1 to 99d40ad3a9

2024-06-03 11:57:25 +02:00

Compare

Weizhen Huang force-pushed restir from 99d40ad3a9 to 95906c5712

2024-06-06 16:47:33 +02:00

Compare

This pull request is marked as a work in progress.

View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u restir:weizhen-restir

git checkout weizhen-restir

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

WIP: Cycles: Importance Resampling #121023

Area ReSTIR: Resampling for Real-Time Defocus and Antialiasing

Checkout