EEVEE-Next Irradiance Baking #105643

New Issue

Clément Foucault · 2023-03-10T19:47:02+01:00

Clément Foucault commented

2023-03-10 19:47:02 +01:00

Due to time constraints it isn't possible to start implementing a solution that leverage raytracing. So the implementation will try to minimize the amount of changes to the current system while cleaning up the implementation and changing the baking algorithm.

Here are the reference papers the new solution will be based on:
https://advances.realtimerendering.com/s2022/SIGGRAPH2022-Advances-Enemies-Ciardi%20et%20al.pdf (Probe-based lighting section)
https://ubm-twvideo01.s3.amazonaws.com/o1/vault/gdc2019/presentations/Hobson_Josh_The_Indirect_Lighting.pdf

~~One remaining unknown is whether or not we keep the light-grids behavior as it was in the original EEVEE or if we go for a single large light-grid structure.~~ It was decided to use volume composition for now.

Here is a breakdown of the tasks:

#105802 Add debug visuals for surfels.
Port Irradiance Cache display.
Port Spherical Harmonics lib & BSDF lib and add unit tests.

Milestone: Dummy irradiance grid displaying.

Rewrite cache structure load / store.
Rewrite light-cache baking in C++ with new draw-manager and unit-tests.

Milestone: Baking, Saving to and Loading from disk should work.

Make shader variation to output surfel.
Bin surfels to ray tiles.
Sort surfels in ray tiles, dead simple sorting for now.
Traverse the rays and add contribution to nearest surfels.
Insert irradiance samples to ray list.
Project result to irradiance grid.

Milestone: Irradiance baking fully works.

Add indirect diffuse contribution to BSDFs
#110386 Add dilation for invalid lightprobes
#110312 Add sampling biases
#110851 Add leak prevention
#106449 Finalize the design and implement the behavior of multiple irradiance grid.

Bugs / Cleanups

Add memory budget option
#113509 Add safety check for out of memory cases during baking
#110858 Add back intensity option
#110858 Add back clamp option to avoid fireflies
Add smooth transition between irradiance volumes
Fix append / link of baked data
Add back capture bounds parameters
Add option on light and object to exclude from irradiance cache
#113362 Add option on material to be considered double sided by irradiance bake
#114176 Port viewport display

Milestone: Irradiance bake minimum-viable-product for 4.0 release.

Extended targets

Support transparency
Support translucency
Support volumes during baking
Sky visibility baking
Auto Sample Resolution based on density
Sparse grid
Light injection
Multi-LOD grid

Dropped features (not considered regressions)

Capture Clip Near / Far distances: Capture is done for the whole scene.
Bounce Count: Light transport is done using an infinite number of bounce and converge to the correct solution with more added samples.

Due to time constraints it isn't possible to start implementing a solution that leverage raytracing. So the implementation will try to minimize the amount of changes to the current system while cleaning up the implementation and changing the baking algorithm. Here are the reference papers the new solution will be based on: https://advances.realtimerendering.com/s2022/SIGGRAPH2022-Advances-Enemies-Ciardi%20et%20al.pdf (Probe-based lighting section) https://ubm-twvideo01.s3.amazonaws.com/o1/vault/gdc2019/presentations/Hobson_Josh_The_Indirect_Lighting.pdf ~~One remaining unknown is whether or not we keep the light-grids behavior as it was in the original EEVEE or if we go for a single large light-grid structure.~~ It was decided to use volume composition for now. Here is a breakdown of the tasks: - [x] #105802 Add debug visuals for surfels. - [x] Port Irradiance Cache display. - [x] Port Spherical Harmonics lib & BSDF lib and add unit tests. #### Milestone: Dummy irradiance grid displaying. - [x] Rewrite cache structure load / store. - [x] Rewrite light-cache baking in C++ with new draw-manager and unit-tests. #### Milestone: Baking, Saving to and Loading from disk should work. - [x] Make shader variation to output surfel. - [x] Bin surfels to ray tiles. - [x] Sort surfels in ray tiles, dead simple sorting for now. - [x] Traverse the rays and add contribution to nearest surfels. - [x] Insert irradiance samples to ray list. - [x] Project result to irradiance grid. #### Milestone: Irradiance baking fully works. - [x] Add indirect diffuse contribution to BSDFs - [x] #110386 Add dilation for invalid lightprobes - [x] #110312 Add sampling biases - [x] #110851 Add leak prevention - [x] #106449 Finalize the design and implement the behavior of multiple irradiance grid. #### Bugs / Cleanups - [x] Add memory budget option - [x] #113509 Add safety check for out of memory cases during baking - [x] #110858 Add back intensity option - [x] #110858 Add back clamp option to avoid fireflies - [x] Add smooth transition between irradiance volumes - [x] Fix append / link of baked data - [x] Add back capture bounds parameters - [x] Add option on light and object to exclude from irradiance cache - [x] #113362 Add option on material to be considered double sided by irradiance bake - [x] #114176 Port viewport display #### Milestone: Irradiance bake minimum-viable-product for 4.0 release. #### Extended targets - Support transparency - Support translucency - Support volumes during baking - [x] Sky visibility baking - Auto Sample Resolution based on density - Sparse grid - Light injection - Multi-LOD grid #### Dropped features (not considered regressions) - Capture Clip Near / Far distances: Capture is done for the whole scene. - Bounce Count: Light transport is done using an infinite number of bounce and converge to the correct solution with more added samples.

👀 2 ❤️ 6 🎉 2 🚀 4

Clément Foucault added the

Type

To Do

label 2023-03-10 19:47:02 +01:00

Clément Foucault self-assigned this 2023-03-10 19:47:02 +01:00

Clément Foucault added this to the Viewport & EEVEE project 2023-03-10 19:47:03 +01:00

Clément Foucault added this to the 4.0 milestone 2023-03-10 22:32:20 +01:00

Clément Foucault referenced this issue

2023-03-12 17:14:29 +01:00

EEVEE-Next #93220

ColumbusUtrigas commented

2023-03-31 01:29:01 +02:00

Hi @fclem
Is it going to be spherical-harmonic-only implementation? If so, is there any plans on opening these structures to Python bindings API? It would be very useful to have full access at least to baked data, ideally being able to manipulate spherical harmonics.

Also, I had plans to add this feature myself but there is a big refactoring going on, so don't want to break it.

Hi @fclem Is it going to be spherical-harmonic-only implementation? If so, is there any plans on opening these structures to Python bindings API? It would be very useful to have full access at least to baked data, ideally being able to manipulate spherical harmonics. Also, I had plans to add this feature myself but there is a big refactoring going on, so don't want to break it.

Emi Martinez commented

2023-03-31 02:16:47 +02:00

Another question @fclem to clarify,
On the Hobson_Josh_The_Indirect_Lighting.pdf talks about using the surfels to bake in real time the light changes, making the Global Illumination dynamic.
But the SIGGRAPH2022-Advances-Enemies-Ciardi20et20al.pdf look like this bake is static, does no react to the light changes.
Will this global illumination be dynamic in real time?

Another question @fclem to clarify, On the Hobson_Josh_The_Indirect_Lighting.pdf talks about using the surfels to bake in real time the light changes, making the Global Illumination dynamic. But the SIGGRAPH2022-Advances-Enemies-Ciardi20et20al.pdf look like this bake is static, does no react to the light changes. Will this global illumination be dynamic in real time?

Clément Foucault commented

2023-03-31 23:09:04 +02:00

Is it going to be spherical-harmonic-only implementation?

Yes. It will have the drawback of SH L1 band not being able to encode opposing lights. Maybe we will expose a L2 option if it isn't too cumbersome.

If so, is there any plans on opening these structures to Python bindings API?

I am considering putting the baked data inside the object or the object-data itself. At that point exposing it through python should be a formality.

Also, I had plans to add this feature myself but there is a big refactoring going on, so don't want to break it.

Sure, better wait until we finalize the baked data design.

Will this global illumination be dynamic in real time?

No. The baking process will be much faster with real-time feedback. But it won't react to real-time light changes.

> Is it going to be spherical-harmonic-only implementation? Yes. It will have the drawback of SH L1 band not being able to encode opposing lights. Maybe we will expose a L2 option if it isn't too cumbersome. > If so, is there any plans on opening these structures to Python bindings API? I am considering putting the baked data inside the object or the object-data itself. At that point exposing it through python should be a formality. > Also, I had plans to add this feature myself but there is a big refactoring going on, so don't want to break it. Sure, better wait until we finalize the baked data design. > Will this global illumination be dynamic in real time? No. The baking process will be much faster with real-time feedback. But it won't react to real-time light changes.

❤️ 1

ColumbusUtrigas commented

2023-04-01 16:54:57 +02:00

In the current implementation Eevee supports both SH2 and HL2 irradiance probes, it looks like it prefers HL2 method and stores all the data in texture array (not sure though how is it stored in DNA system, it just serializes this Image as is?). Isn't it replicated somewhere in RAM?

You said there will be only SH (1?) implementation, will storage method be the same image array in LightProbe datatype? If there will be both SH1 and SH2 implementations, how they will be distincted on a data level?

In the current implementation Eevee supports both SH2 and HL2 irradiance probes, it looks like it prefers HL2 method and stores all the data in texture array (not sure though how is it stored in DNA system, it just serializes this Image as is?). Isn't it replicated somewhere in RAM? You said there will be only SH (1?) implementation, will storage method be the same image array in LightProbe datatype? If there will be both SH1 and SH2 implementations, how they will be distincted on a data level?

Clément Foucault commented

2023-04-01 17:26:12 +02:00

@ColumbusUtrigas

In the current implementation Eevee supports both SH2 and HL2 irradiance probes

Well, yes, but only HL2 is used.

[...] not sure though how is it stored in DNA system, it just serializes this Image as is?

That is correct. We use an atlas for all the probe.

will storage method be the same image array in LightProbe datatype?

No. We will leverage the benefits of 3D volume textures for the new implementation. And each grid will have its own storage instead of having a common atlas. The layout will be as described in the first reference. However if we expose it to python I expect we do something better than the packed data and return a full float representation per SH band.

@ColumbusUtrigas > In the current implementation Eevee supports both SH2 and HL2 irradiance probes Well, yes, but only HL2 is used. > [...] not sure though how is it stored in DNA system, it just serializes this Image as is? That is correct. We use an atlas for all the probe. > will storage method be the same image array in LightProbe datatype? No. We will leverage the benefits of 3D volume textures for the new implementation. And each grid will have its own storage instead of having a common atlas. The layout will be as described in the first reference. However if we expose it to python I expect we do something better than the packed data and return a full float representation per SH band.

ColumbusUtrigas commented

2023-04-01 17:52:28 +02:00

The layout will be as described in the first reference

Is data layout already known? And overall, what time estimates/deadlines are on this part of eevee? Any help needed? Maybe some work needed on a Vulkan backend and VK RT support?

However if we expose it to python I expect we do something better than the packed data

How do you plan to pack SH2 into a 3D texture?

and return a full float representation per SH band

You mean being able to return both SH1 and SH2 full representation per probe?

As I see it:

bpy.data.lightprobes['Irradiance Probe'].spherical_harmonics2 # returns array of SH2, may be packed if packing algorithm is known, but ideally is a 3d grid with 9 coefficients per cell (4d array)

sh = bpy.data.lightprobes['Irradiance Probe'].get_sh2([1,2,3]) # parameter is a 3d cell id
irradiance = sh.evaluate(normal) # optional

# or

irradiance = bpy.data.lightprobes['Irradiance Probe'].evaluate_grid(point, normal) # optional

I think it's not needed to be able to modify probes as they may be rebaked soon anyway, but even if it's needed it is easy to implement on top.

> The layout will be as described in the first reference Is data layout already known? And overall, what time estimates/deadlines are on this part of eevee? Any help needed? Maybe some work needed on a Vulkan backend and VK RT support? > However if we expose it to python I expect we do something better than the packed data How do you plan to pack SH2 into a 3D texture? > and return a full float representation per SH band You mean being able to return both SH1 and SH2 full representation per probe? As I see it: ```Python bpy.data.lightprobes['Irradiance Probe'].spherical_harmonics2 # returns array of SH2, may be packed if packing algorithm is known, but ideally is a 3d grid with 9 coefficients per cell (4d array) sh = bpy.data.lightprobes['Irradiance Probe'].get_sh2([1,2,3]) # parameter is a 3d cell id irradiance = sh.evaluate(normal) # optional # or irradiance = bpy.data.lightprobes['Irradiance Probe'].evaluate_grid(point, normal) # optional ``` I think it's not needed to be able to modify probes as they may be rebaked soon anyway, but even if it's needed it is easy to implement on top.

Clément Foucault commented

2023-04-02 12:47:38 +02:00

@ColumbusUtrigas Yes the API would look more or less like that. I have created #106449 to discuss all of this into more detail.

Is data layout already known?

That's a good question. I would say no, because I would like to have different layout on the storage side (for efficient loading & modification) than on the runtime side (for more efficient shading & VRAM usage).

And overall, what time estimates/deadlines are on this part of eevee?

EEVEE-Next is scheduled for 4.0. But some of what we discussed might be considered extended goals.

Any help needed? Maybe some work needed on a Vulkan backend and VK RT support?

@Jeroen-Bakker Is doing the vulkan port. If you want to contribute, you can join the #eevee-viewport-module and ask for suggestions.

@ColumbusUtrigas Yes the API would look more or less like that. I have created #106449 to discuss all of this into more detail. > Is data layout already known? That's a good question. I would say no, because I would like to have different layout on the storage side (for efficient loading & modification) than on the runtime side (for more efficient shading & VRAM usage). > And overall, what time estimates/deadlines are on this part of eevee? EEVEE-Next is scheduled for 4.0. But some of what we discussed might be considered extended goals. > Any help needed? Maybe some work needed on a Vulkan backend and VK RT support? @Jeroen-Bakker Is doing the vulkan port. If you want to contribute, you can join the [#eevee-viewport-module](https://blender.chat/channel/eevee-viewport-module) and ask for suggestions.

👍 3

Clément Foucault referenced this issue

2023-06-06 00:59:27 +02:00

EEVEE-Next: Irradiance Cache: Initial Implementation #108639

Clément Foucault referenced this issue from a commit

2023-06-23 08:39:54 +02:00

EEVEE-Next: Irradiance Cache: Initial Implementation

Clément Foucault added the