Eevee-next: Reflection Probe Packing #109688

Merged
Jeroen Bakker merged 34 commits from Jeroen-Bakker/blender:eevee-next-reflection-probe-packing into main 2023-07-07 15:37:35 +02:00
Member

All probes (including the world background probe) are stored in a single texture. Each probe
can be of any resolution as long as it is a power of 2 and not larger than 2048. So valid options
are (2048x2048, 1024x1024, 512x512, etc).

Each probe can be stored in their own resolution and can be set by the user.

NOTE: Eventually we would like to add automatic resolution selection.

The probes are packed in an 2d texture array with the dimension of 2048*2048. The number of
layers depends on the actual needed layers. If more layers are needed the texture will be recreated.
This can happen when a new reflection probe is added, or an existing reflection probe is made visible
to the scene or its resolution is changed.

Octahedral mapping

Probes are rendered into a cubemap. To reduce memory needs and improve sampling performance the cubemap
is stored in octahedral mapping space. This is done in eevee_reflection_probe_remap_comp.glsl.

The regular octahedral mapping has been extended to fix leakages at the edges of the texture
and to be able to be used on an atlas texture and by sampling the texture once.

To reduce sampling cost and improve the quality we add an border around the
octahedral map and extend the octahedral coordinates. This also allows us to
generate lower resolution mipmaps of the atlas texture using 2x2 box filtering
from a higher resolution.

Subdivisions and areas

Probes data are stored besides the texture. The data is used to find out where the probe is stored
in the texture. It is also used to find free space to store new probes.

For each probe the next data is stored:

  • layer: on which layer of the texture contains the data of the probe
  • layer_subdivision: what subdivision is used by the probe. With the subdivision the resolution of
    probe in the layer can be determined. 0 would use the resolution of the texture (2048x2048). 1 divides the texture in 4 even areas of (1024x1024). 2 divides the texture in 16 areas of (512x512).
  • area_index: What area of the layer contains the probe. A layer contains out of (2^layer_subdivision) * (2^layer_subdivision) areas.

This approach ensures that we can be flexible at storing probes with different
resolutions on the same layer. Lets see an example how that works

Example

Lets assume we have a world probe with the resolution of 1024x1024 and 2 reflection probes
with the resolution of 512x512. The resolution of the texture is 2048x2048. This could be
stored in the texture as follow

Probe layer layer_subdivision area_index Area in texture coordinates
0 0 1 0 0,0,0 - 1023,1023,0
1 0 2 2 1024,0,0 - 1535,511,0
2 0 2 3 1536,0,0 - 2047,511,0

Visually:

Layer 0 viewed at subdivision level 2
+---+---+---+---+
|   |   |   |   |
|   |   |   |   |
+---+---+---+---+
|   |   |   |   |
|   |   |   |   |
+---+---+---+---+
|000|000|   |   |
|000|000|   |   |
+---+---+---+---+
|000|000|111|222|
|000|000|111|222|
+---+---+---+---+

Looking at the same data, but in subdivision level 1 would look like:

Layer 0 viewed at subdivision level 1
+-------+-------+
|       |       |
|       |       |
|       |       |
|       |       |
|       |       |
+-------+-------+
|0000000|1212121|
|0000000|2121212|
|0000000|1212121|
|0000000|2121212|
|0000000|1212121|
+-------+-------+

On subdivision level 1 there are 2 empty spaces left, the lower right part is allocated by probes with a higher subdivision. When wanting to allocate an area with subdivision 1 it would select area 0,1024,0 - 1023,2047,0.

Code-wise this is implemented by ProbeLocationFinder. ProbeLocationFinder can view a texture in a
given subdivision level and mark areas that are covered by probes. When finding a free spot it returns
the first empty area.

Notes

  • Currently the cubemap is rendered with a fixed resolution and mipmaps are generated in order to increase the quality of the atlas. Eventually we should use dynamic resolution and no mipmaps. This will be done as part of the light probe baking change.
All probes (including the world background probe) are stored in a single texture. Each probe can be of any resolution as long as it is a power of 2 and not larger than 2048. So valid options are (2048x2048, 1024x1024, 512x512, etc). Each probe can be stored in their own resolution and can be set by the user. > NOTE: Eventually we would like to add automatic resolution selection. The probes are packed in an 2d texture array with the dimension of 2048*2048. The number of layers depends on the actual needed layers. If more layers are needed the texture will be recreated. This can happen when a new reflection probe is added, or an existing reflection probe is made visible to the scene or its resolution is changed. ### Octahedral mapping Probes are rendered into a cubemap. To reduce memory needs and improve sampling performance the cubemap is stored in octahedral mapping space. This is done in `eevee_reflection_probe_remap_comp.glsl`. The regular octahedral mapping has been extended to fix leakages at the edges of the texture and to be able to be used on an atlas texture and by sampling the texture once. To reduce sampling cost and improve the quality we add an border around the octahedral map and extend the octahedral coordinates. This also allows us to generate lower resolution mipmaps of the atlas texture using 2x2 box filtering from a higher resolution. ### Subdivisions and areas Probes data are stored besides the texture. The data is used to find out where the probe is stored in the texture. It is also used to find free space to store new probes. For each probe the next data is stored: * **layer**: on which layer of the texture contains the data of the probe * **layer_subdivision**: what subdivision is used by the probe. With the subdivision the resolution of probe in the layer can be determined. 0 would use the resolution of the texture (2048x2048). 1 divides the texture in 4 even areas of (1024x1024). 2 divides the texture in 16 areas of (512x512). * **area_index**: What area of the layer contains the probe. A layer contains out of `(2^layer_subdivision) * (2^layer_subdivision)` areas. This approach ensures that we can be flexible at storing probes with different resolutions on the same layer. Lets see an example how that works ### Example Lets assume we have a world probe with the resolution of 1024x1024 and 2 reflection probes with the resolution of 512x512. The resolution of the texture is 2048x2048. This could be stored in the texture as follow | Probe | layer | layer_subdivision | area_index | Area in texture coordinates | |-------|-------|-------------------|------------|-----------------------------| | 0 | 0 | 1 | 0 | 0,0,0 - 1023,1023,0 | | 1 | 0 | 2 | 2 | 1024,0,0 - 1535,511,0 | | 2 | 0 | 2 | 3 | 1536,0,0 - 2047,511,0 | Visually: ``` Layer 0 viewed at subdivision level 2 +---+---+---+---+ | | | | | | | | | | +---+---+---+---+ | | | | | | | | | | +---+---+---+---+ |000|000| | | |000|000| | | +---+---+---+---+ |000|000|111|222| |000|000|111|222| +---+---+---+---+ ``` Looking at the same data, but in subdivision level 1 would look like: ``` Layer 0 viewed at subdivision level 1 +-------+-------+ | | | | | | | | | | | | | | | +-------+-------+ |0000000|1212121| |0000000|2121212| |0000000|1212121| |0000000|2121212| |0000000|1212121| +-------+-------+ ``` On subdivision level 1 there are 2 empty spaces left, the lower right part is allocated by probes with a higher subdivision. When wanting to allocate an area with subdivision 1 it would select area 0,1024,0 - 1023,2047,0. Code-wise this is implemented by `ProbeLocationFinder`. ProbeLocationFinder can view a texture in a given subdivision level and mark areas that are covered by probes. When finding a free spot it returns the first empty area. **Notes** * Currently the cubemap is rendered with a fixed resolution and mipmaps are generated in order to increase the quality of the atlas. Eventually we should use dynamic resolution and no mipmaps. This will be done as part of the light probe baking change.
Jeroen Bakker added this to the 4.0 milestone 2023-07-04 10:35:15 +02:00
Jeroen Bakker added the
Module
EEVEE & Viewport
label 2023-07-04 10:35:15 +02:00
Jeroen Bakker self-assigned this 2023-07-04 10:35:15 +02:00
Jeroen Bakker added 6 commits 2023-07-04 10:35:26 +02:00
Jeroen Bakker added 1 commit 2023-07-04 13:04:06 +02:00
Jeroen Bakker added 1 commit 2023-07-04 13:53:41 +02:00
Jeroen Bakker added 1 commit 2023-07-04 14:16:44 +02:00
Jeroen Bakker changed title from WIP: Eevee-next: Reflection Probe Packing to Eevee-next: Reflection Probe Packing 2023-07-04 14:17:31 +02:00
Jeroen Bakker added this to the EEVEE & Viewport project 2023-07-04 14:21:01 +02:00
Jeroen Bakker requested review from Clément Foucault 2023-07-04 14:21:07 +02:00
Jeroen Bakker added 1 commit 2023-07-04 16:03:25 +02:00
Jeroen Bakker added 1 commit 2023-07-04 16:05:46 +02:00
Jeroen Bakker reviewed 2023-07-06 09:29:57 +02:00
@ -73,1 +75,4 @@
/* LightProbe->bake_resolution, World->bake_resolution. */
typedef enum eLightProbeBakeResolution {
LIGHT_PROBE_BAKE_RESOLUTION_64 = 6,
Author
Member

Consider removal or using less mipmap levels for small sizes.

Consider removal or using less mipmap levels for small sizes.
Jeroen-Bakker marked this conversation as resolved
Jeroen Bakker added 2 commits 2023-07-06 12:26:09 +02:00
Jeroen Bakker added 1 commit 2023-07-06 12:43:42 +02:00
Jeroen Bakker added 1 commit 2023-07-06 12:57:28 +02:00
Jeroen Bakker added 3 commits 2023-07-06 14:53:00 +02:00
Jeroen Bakker added 1 commit 2023-07-06 15:12:14 +02:00
Jeroen Bakker added 1 commit 2023-07-06 15:22:07 +02:00
Jeroen Bakker added 1 commit 2023-07-06 15:41:43 +02:00
Clément Foucault requested changes 2023-07-06 15:57:20 +02:00
@ -1025,0 +1038,4 @@
* TODO: remove this attribute when baking has been implemented.
* NOTE: alpha channel is not used, but here for alignment reasons.
*/
float4 color;

Use random color instead of adding dummy member to the struct

Use random color instead of adding dummy member to the struct
Jeroen-Bakker marked this conversation as resolved
@ -1025,0 +1041,4 @@
float4 color;
packed_float3 pos;
/** On which layer of the cubemaps array is this reflection probe stored. */

Update documentation. Not a cubemap anymore

Update documentation. Not a cubemap anymore
Jeroen-Bakker marked this conversation as resolved
@ -1025,0 +1067,4 @@
/**
* Artistic control to change the intensity during evaluation phase.
*/
float intensity;

Remove that option for now.

Remove that option for now.
Jeroen-Bakker marked this conversation as resolved
@ -10,14 +10,23 @@
* \{ */
GPU_SHADER_CREATE_INFO(eevee_reflection_probe_data)
.storage_buf(REFLECTION_PROBE_BUF_SLOT,

Consider using UBO instead.

Consider using UBO instead.
Jeroen-Bakker marked this conversation as resolved
Jeroen Bakker added 1 commit 2023-07-06 16:57:01 +02:00
Jeroen Bakker reviewed 2023-07-06 20:02:48 +02:00
Jeroen Bakker reviewed 2023-07-06 20:03:09 +02:00
@ -74,0 +81,4 @@
LIGHT_PROBE_BAKE_RESOLUTION_512 = 9,
LIGHT_PROBE_BAKE_RESOLUTION_1024 = 10,
LIGHT_PROBE_BAKE_RESOLUTION_2048 = 11,
LIGHT_PROBE_BAKE_RESOLUTION_4096 = 12,
Author
Member

Remove item

Remove item
Jeroen-Bakker marked this conversation as resolved
Jeroen Bakker reviewed 2023-07-06 20:03:53 +02:00
@ -34,0 +73,4 @@
(1.0 - 2.0 * REFLECTION_PROBE_BORDER_SIZE * texel_size);
/* Mirror until the coordinates fit. */
/* Fix right side. */
Author
Member

Can we use a checkerboard to reduce the branches even more?

We could monitor the delta octahedral uvs

Can we use a checkerboard to reduce the branches even more? We could monitor the delta octahedral uvs
Jeroen-Bakker marked this conversation as resolved
Clément Foucault requested changes 2023-07-06 22:25:52 +02:00
@ -289,0 +296,4 @@
if (!DNA_struct_elem_find(fd->filesdna, "World", "int", "bake_resolution")) {
LISTBASE_FOREACH (World *, world, &bmain->worlds) {
world->bake_resolution = LIGHT_PROBE_BAKE_RESOLUTION_1024;

I wouldn't call it bake_resolution because it isn't necessarily related to baking.
Maybe probe_resolution is better for world and resolution for lightprobes.

I wouldn't call it `bake_resolution` because it isn't necessarily related to baking. Maybe `probe_resolution` is better for world and `resolution` for lightprobes.
Jeroen-Bakker marked this conversation as resolved
@ -30,1 +30,4 @@
#define REFLECTION_PROBES_MAX 256
#define REFLECTION_PROBE_GROUP_SIZE 16
/* Number of additional pixels on the border of an octahedral map to reserve for fixing leakage.*/
#define REFLECTION_PROBE_BORDER_SIZE 16.0

make it float(1 << (REFLECTION_PROBE_MIPMAP_LEVELS - 1))

make it `float(1 << (REFLECTION_PROBE_MIPMAP_LEVELS - 1))`
Jeroen-Bakker marked this conversation as resolved
@ -1025,0 +1027,4 @@
* \{ */
/** Mapping data to locate a reflection probe in texture. */
struct ReflectionProbeData {

Might be left to another PR, but one parameter that was omitted is the parallax shape and distance. Unfortunately, this means increasing the storage quite a bit to have a 3x4 matrix.

Might be left to another PR, but one parameter that was omitted is the parallax shape and distance. Unfortunately, this means increasing the storage quite a bit to have a 3x4 matrix.
fclem marked this conversation as resolved
@ -1025,0 +1061,4 @@
};
BLI_STATIC_ASSERT_ALIGN(ReflectionProbeData, 16)
struct ReflectionProbeBuffer {

Do not use a wrapper struct. Use UniformArrayBuffer<<ReflectionProbeData, REFLECTION_PROBES_MAX>

Do not use a wrapper struct. Use `UniformArrayBuffer<<ReflectionProbeData, REFLECTION_PROBES_MAX>`
Jeroen-Bakker marked this conversation as resolved
Jeroen Bakker added 1 commit 2023-07-07 07:52:32 +02:00
Jeroen Bakker added 1 commit 2023-07-07 08:35:27 +02:00
Jeroen Bakker added 1 commit 2023-07-07 08:38:11 +02:00
Jeroen Bakker added 4 commits 2023-07-07 09:04:08 +02:00
Clément Foucault requested changes 2023-07-07 12:05:17 +02:00
@ -34,2 +59,4 @@
pass.bind_texture("cubemap_tx", cubemap_tx_);
pass.bind_image("octahedral_img", probes_tx_);
pass.bind_ssbo(REFLECTION_PROBE_BUF_SLOT, data_buf_);
/* TODO this should be added in a subpass with the correct resolution to reduce overshooting

you can use pass.dispatch(&dispatch_probe_pack_) and set it inside remap_to_octahedral_projection.

you can use `pass.dispatch(&dispatch_probe_pack_)` and set it inside `remap_to_octahedral_projection`.
Jeroen-Bakker marked this conversation as resolved
@ -39,0 +188,4 @@
}
/**
* Utility class to find a location in the cubemap that can be used to store a new probe cubemap in

Check for all occurences of cubemap in the comments and see if you can change it to texture or probe.

Check for all occurences of cubemap in the comments and see if you can change it to `texture` or `probe`.
Jeroen-Bakker marked this conversation as resolved
@ -39,0 +301,4 @@
bool regenerate_mipmaps_postponed = false;
for (ReflectionProbe &cubemap : probes_) {
cubemap.is_dirty |= resize_layers;
if (resize_layers) {
    cubemap.is_dirty = true;
    cubemap.is_probes_tx_dirty = true;
}

Having a bitwise operator on a bool feels weird.

``` if (resize_layers) { cubemap.is_dirty = true; cubemap.is_probes_tx_dirty = true; } ``` Having a bitwise operator on a bool feels weird.
Jeroen-Bakker marked this conversation as resolved
@ -39,0 +384,4 @@
}
/* -------------------------------------------------------------------- */
/** \name World

Wrong naming. also maybe leave ReflectionProbe::needs_update() inside the header.

Wrong naming. also maybe leave `ReflectionProbe::needs_update()` inside the header.
Jeroen-Bakker marked this conversation as resolved
@ -28,0 +32,4 @@
Type type = Type::Unused;
bool is_dirty = false;

It isn't clear what is the difference between is_dirty and is_probes_tx_dirty.
As far as I understand, is_probes_tx_dirty is true when mips needs to be reconstructed and is_dirty when it needs to be rerendered.
So maybe do_render and do_mipmap or update_render and update_mipmap.
Either way, their comments should be clearer.

It isn't clear what is the difference between `is_dirty` and `is_probes_tx_dirty`. As far as I understand, `is_probes_tx_dirty` is true when mips needs to be reconstructed and `is_dirty` when it needs to be rerendered. So maybe `do_render` and `do_mipmap` or `update_render` and `update_mipmap`. Either way, their comments should be clearer.
Jeroen-Bakker marked this conversation as resolved
@ -28,0 +53,4 @@
*/
int index = -1;
bool needs_update() const;

Document function. It's not visible right away why would you need this over is_dirty when both are public.

Document function. It's not visible right away why would you need this over `is_dirty` when both are public.
Jeroen-Bakker marked this conversation as resolved
@ -39,2 +70,4 @@
Instance &instance_;
ReflectionProbeDataBuf data_buf_;
Vector<ReflectionProbe> probes_;

Why use a Vector here and not a Map<ObjectKey, ReflectionProbe> like other modules? The linear search in find_or_insert is a bit scary and seems to reimplement some logic already present in Map.

Why use a `Vector` here and not a `Map<ObjectKey, ReflectionProbe>` like other modules? The linear search in `find_or_insert` is a bit scary and seems to reimplement some logic already present in `Map`.
Jeroen-Bakker marked this conversation as resolved
@ -215,6 +215,7 @@ void CaptureView::render()
view.sync(view_m4, win_m4);
inst_.pipelines.world.render(view);
}
GPU_texture_update_mipmap_chain(inst_.reflection_probes.cubemap_tx_);

Do you really need this since you create the LOD of the octahedral map?

Do you really need this since you create the LOD of the octahedral map?
Author
Member

The cubemap has a fix size of 1024x1024. When remapping a probe of 64x64 we want to use a different mipmap level so the light is more evenly spread. We could remove the mipmap and use a dynamic cubemap size. or leave it how it is now.

I do think the resolution should become dynamic, but would postpone that for the light probe rendering part. I will mention that in the PR description

The cubemap has a fix size of 1024x1024. When remapping a probe of 64x64 we want to use a different mipmap level so the light is more evenly spread. We could remove the mipmap and use a dynamic cubemap size. or leave it how it is now. I do think the resolution should become dynamic, but would postpone that for the light probe rendering part. I will mention that in the PR description
Jeroen-Bakker marked this conversation as resolved
@ -58,3 +64,1 @@
* the reflection probe patch. */
const float intensity_factor = 1.0;
out_specular += vec3(intensity_factor * l_col);
/* ReflectionProbeData doesn't contain any gabs, exit at first item that is invalid. */

Typo: gabs > gap

Typo: gabs > gap
Jeroen-Bakker marked this conversation as resolved
Jeroen Bakker added 1 commit 2023-07-07 12:37:23 +02:00
Clément Foucault reviewed 2023-07-07 12:39:01 +02:00
@ -28,0 +45,4 @@
/**
* Probes that aren't used during a draw can be cleared.
*/
bool is_used = false;

How is that not redundant with type == Type::Unused ?

How is that not redundant with `type == Type::Unused` ?
Author
Member

Isn't the same. Shoudl clarify that this is for Type::Probe to detect if the probe is still in use.

Isn't the same. Shoudl clarify that this is for Type::Probe to detect if the probe is still in use.
Jeroen-Bakker marked this conversation as resolved
Jeroen Bakker added 2 commits 2023-07-07 14:24:53 +02:00
Clément Foucault approved these changes 2023-07-07 14:27:16 +02:00
Author
Member

@blender-bot build

@blender-bot build
Jeroen Bakker added 2 commits 2023-07-07 14:57:56 +02:00
Author
Member

@blender-bot build

@blender-bot build
Jeroen Bakker merged commit 17a58f7db0 into main 2023-07-07 15:37:34 +02:00
Jeroen Bakker deleted branch eevee-next-reflection-probe-packing 2023-07-07 15:37:36 +02:00
Sign in to join this conversation.
No reviewers
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#109688
No description provided.