EEVEE-Next Irradiance Baking #105643

New Issue

Due to time constraints it isn't possible to start implementing a solution that leverage raytracing. So the implementation will try to minimize the amount of changes to the current system while cleaning up the implementation and changing the baking algorithm.

Here are the reference papers the new solution will be based on:
https://advances.realtimerendering.com/s2022/SIGGRAPH2022-Advances-Enemies-Ciardi%20et%20al.pdf (Probe-based lighting section)
https://ubm-twvideo01.s3.amazonaws.com/o1/vault/gdc2019/presentations/Hobson_Josh_The_Indirect_Lighting.pdf

One remaining unknown is whether or not we keep the light-grids behavior as it was in the original EEVEE or if we go for a single large light-grid structure.

Here is a breakdown of the tasks:

#105802 Add debug visuals for surfels.
Port Irradiance Cache display.
Port Spherical Harmonics lib & BSDF lib and add unit tests.

Milestone: Dummy irradiance grid displaying.

Rewrite cache structure load / store.
Rewrite light-cache baking in C++ with new draw-manager and unit-tests.

Milestone: Baking, Saving to and Loading from disk should work.

Make shader variation to output surfel.
Bin surfels to ray tiles.
Sort surfels in ray tiles, dead simple sorting for now.
Traverse the rays and add contribution to nearest surfels.
Insert irradiance samples to ray list.
Project result to irradiance grid.

Milestone: Irradiance baking fully works.

Add indirect diffuse contribution to BSDFs
Add dilation for invalid lightprobes
Add leak prevention
#106449 Finalize the design and implement the behavior of multiple irradiance grid.

Milestone: Irradiance bake minimum-viable-product for 4.0 release.

Extended targets

Support transparency
Support translucency
Sky visibility baking
Sparse grid
Multi-LOD grid

Due to time constraints it isn't possible to start implementing a solution that leverage raytracing. So the implementation will try to minimize the amount of changes to the current system while cleaning up the implementation and changing the baking algorithm. Here are the reference papers the new solution will be based on: https://advances.realtimerendering.com/s2022/SIGGRAPH2022-Advances-Enemies-Ciardi%20et%20al.pdf (Probe-based lighting section) https://ubm-twvideo01.s3.amazonaws.com/o1/vault/gdc2019/presentations/Hobson_Josh_The_Indirect_Lighting.pdf One remaining unknown is whether or not we keep the light-grids behavior as it was in the original EEVEE or if we go for a single large light-grid structure. Here is a breakdown of the tasks: - [x] #105802 Add debug visuals for surfels. - [x] Port Irradiance Cache display. - [x] Port Spherical Harmonics lib & BSDF lib and add unit tests. #### Milestone: Dummy irradiance grid displaying. - [x] Rewrite cache structure load / store. - [x] Rewrite light-cache baking in C++ with new draw-manager and unit-tests. #### Milestone: Baking, Saving to and Loading from disk should work. - [x] Make shader variation to output surfel. - [x] Bin surfels to ray tiles. - [x] Sort surfels in ray tiles, dead simple sorting for now. - [x] Traverse the rays and add contribution to nearest surfels. - [x] Insert irradiance samples to ray list. - [x] Project result to irradiance grid. #### Milestone: Irradiance baking fully works. - [ ] Add indirect diffuse contribution to BSDFs - [ ] Add dilation for invalid lightprobes - [ ] Add leak prevention - [ ] #106449 Finalize the design and implement the behavior of multiple irradiance grid. #### Milestone: Irradiance bake minimum-viable-product for 4.0 release. #### Extended targets - [ ] Support transparency - [ ] Support translucency - [ ] Sky visibility baking - [ ] Sparse grid - [ ] Multi-LOD grid

Hi @fclem
Is it going to be spherical-harmonic-only implementation? If so, is there any plans on opening these structures to Python bindings API? It would be very useful to have full access at least to baked data, ideally being able to manipulate spherical harmonics.

Also, I had plans to add this feature myself but there is a big refactoring going on, so don't want to break it.

Hi @fclem Is it going to be spherical-harmonic-only implementation? If so, is there any plans on opening these structures to Python bindings API? It would be very useful to have full access at least to baked data, ideally being able to manipulate spherical harmonics. Also, I had plans to add this feature myself but there is a big refactoring going on, so don't want to break it.

Another question @fclem to clarify,
On the Hobson_Josh_The_Indirect_Lighting.pdf talks about using the surfels to bake in real time the light changes, making the Global Illumination dynamic.
But the SIGGRAPH2022-Advances-Enemies-Ciardi20et20al.pdf look like this bake is static, does no react to the light changes.
Will this global illumination be dynamic in real time?

Another question @fclem to clarify, On the Hobson_Josh_The_Indirect_Lighting.pdf talks about using the surfels to bake in real time the light changes, making the Global Illumination dynamic. But the SIGGRAPH2022-Advances-Enemies-Ciardi20et20al.pdf look like this bake is static, does no react to the light changes. Will this global illumination be dynamic in real time?

Is it going to be spherical-harmonic-only implementation?

Yes. It will have the drawback of SH L1 band not being able to encode opposing lights. Maybe we will expose a L2 option if it isn't too cumbersome.

If so, is there any plans on opening these structures to Python bindings API?

I am considering putting the baked data inside the object or the object-data itself. At that point exposing it through python should be a formality.

Also, I had plans to add this feature myself but there is a big refactoring going on, so don't want to break it.

Sure, better wait until we finalize the baked data design.

Will this global illumination be dynamic in real time?

No. The baking process will be much faster with real-time feedback. But it won't react to real-time light changes.

> Is it going to be spherical-harmonic-only implementation? Yes. It will have the drawback of SH L1 band not being able to encode opposing lights. Maybe we will expose a L2 option if it isn't too cumbersome. > If so, is there any plans on opening these structures to Python bindings API? I am considering putting the baked data inside the object or the object-data itself. At that point exposing it through python should be a formality. > Also, I had plans to add this feature myself but there is a big refactoring going on, so don't want to break it. Sure, better wait until we finalize the baked data design. > Will this global illumination be dynamic in real time? No. The baking process will be much faster with real-time feedback. But it won't react to real-time light changes.

could you add that again? "ShaderToRGB node will remove any SSR and SSS from a shader."
I can't update blender anymore because my models use this system and they look very ugly without them, I beg you to add this again
#104496 (comment)

could you add that again? "ShaderToRGB node will remove any SSR and SSS from a shader." I can't update blender anymore because my models use this system and they look very ugly without them, I beg you to add this again https://projects.blender.org/blender/blender/issues/104496#issue-93465

In the current implementation Eevee supports both SH2 and HL2 irradiance probes, it looks like it prefers HL2 method and stores all the data in texture array (not sure though how is it stored in DNA system, it just serializes this Image as is?). Isn't it replicated somewhere in RAM?

You said there will be only SH (1?) implementation, will storage method be the same image array in LightProbe datatype? If there will be both SH1 and SH2 implementations, how they will be distincted on a data level?

In the current implementation Eevee supports both SH2 and HL2 irradiance probes, it looks like it prefers HL2 method and stores all the data in texture array (not sure though how is it stored in DNA system, it just serializes this Image as is?). Isn't it replicated somewhere in RAM? You said there will be only SH (1?) implementation, will storage method be the same image array in LightProbe datatype? If there will be both SH1 and SH2 implementations, how they will be distincted on a data level?

@ColumbusUtrigas

In the current implementation Eevee supports both SH2 and HL2 irradiance probes

Well, yes, but only HL2 is used.

[...] not sure though how is it stored in DNA system, it just serializes this Image as is?

That is correct. We use an atlas for all the probe.

will storage method be the same image array in LightProbe datatype?

No. We will leverage the benefits of 3D volume textures for the new implementation. And each grid will have its own storage instead of having a common atlas. The layout will be as described in the first reference. However if we expose it to python I expect we do something better than the packed data and return a full float representation per SH band.

@ColumbusUtrigas > In the current implementation Eevee supports both SH2 and HL2 irradiance probes Well, yes, but only HL2 is used. > [...] not sure though how is it stored in DNA system, it just serializes this Image as is? That is correct. We use an atlas for all the probe. > will storage method be the same image array in LightProbe datatype? No. We will leverage the benefits of 3D volume textures for the new implementation. And each grid will have its own storage instead of having a common atlas. The layout will be as described in the first reference. However if we expose it to python I expect we do something better than the packed data and return a full float representation per SH band.

The layout will be as described in the first reference

Is data layout already known? And overall, what time estimates/deadlines are on this part of eevee? Any help needed? Maybe some work needed on a Vulkan backend and VK RT support?

However if we expose it to python I expect we do something better than the packed data

How do you plan to pack SH2 into a 3D texture?

and return a full float representation per SH band

You mean being able to return both SH1 and SH2 full representation per probe?

As I see it:

bpy.data.lightprobes['Irradiance Probe'].spherical_harmonics2 # returns array of SH2, may be packed if packing algorithm is known, but ideally is a 3d grid with 9 coefficients per cell (4d array)

sh = bpy.data.lightprobes['Irradiance Probe'].get_sh2([1,2,3]) # parameter is a 3d cell id
irradiance = sh.evaluate(normal) # optional

# or

irradiance = bpy.data.lightprobes['Irradiance Probe'].evaluate_grid(point, normal) # optional

I think it's not needed to be able to modify probes as they may be rebaked soon anyway, but even if it's needed it is easy to implement on top.

> The layout will be as described in the first reference Is data layout already known? And overall, what time estimates/deadlines are on this part of eevee? Any help needed? Maybe some work needed on a Vulkan backend and VK RT support? > However if we expose it to python I expect we do something better than the packed data How do you plan to pack SH2 into a 3D texture? > and return a full float representation per SH band You mean being able to return both SH1 and SH2 full representation per probe? As I see it: ```Python bpy.data.lightprobes['Irradiance Probe'].spherical_harmonics2 # returns array of SH2, may be packed if packing algorithm is known, but ideally is a 3d grid with 9 coefficients per cell (4d array) sh = bpy.data.lightprobes['Irradiance Probe'].get_sh2([1,2,3]) # parameter is a 3d cell id irradiance = sh.evaluate(normal) # optional # or irradiance = bpy.data.lightprobes['Irradiance Probe'].evaluate_grid(point, normal) # optional ``` I think it's not needed to be able to modify probes as they may be rebaked soon anyway, but even if it's needed it is easy to implement on top.

@ColumbusUtrigas Yes the API would look more or less like that. I have created #106449 to discuss all of this into more detail.

Is data layout already known?

That's a good question. I would say no, because I would like to have different layout on the storage side (for efficient loading & modification) than on the runtime side (for more efficient shading & VRAM usage).

And overall, what time estimates/deadlines are on this part of eevee?

EEVEE-Next is scheduled for 4.0. But some of what we discussed might be considered extended goals.

Any help needed? Maybe some work needed on a Vulkan backend and VK RT support?

@Jeroen-Bakker Is doing the vulkan port. If you want to contribute, you can join the #eevee-viewport-module and ask for suggestions.

@ColumbusUtrigas Yes the API would look more or less like that. I have created #106449 to discuss all of this into more detail. > Is data layout already known? That's a good question. I would say no, because I would like to have different layout on the storage side (for efficient loading & modification) than on the runtime side (for more efficient shading & VRAM usage). > And overall, what time estimates/deadlines are on this part of eevee? EEVEE-Next is scheduled for 4.0. But some of what we discussed might be considered extended goals. > Any help needed? Maybe some work needed on a Vulkan backend and VK RT support? @Jeroen-Bakker Is doing the vulkan port. If you want to contribute, you can join the [#eevee-viewport-module](https://blender.chat/channel/eevee-viewport-module) and ask for suggestions.

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

EEVEE-Next Irradiance Baking #105643

Milestone: Dummy irradiance grid displaying.

Milestone: Baking, Saving to and Loading from disk should work.

Milestone: Irradiance baking fully works.

Milestone: Irradiance bake minimum-viable-product for 4.0 release.

Extended targets