LineArt shadow functionality design notes #98498

New Issue

YimingWu · 2022-05-30T15:55:12+02:00

YimingWu commented

2022-05-30 15:55:12 +02:00

I decide to put the design notes in this separate task for the moment, so this way I can explain things much better, it will be a complement to the shadow patch (yet to come) information.

I will add the corresponding function calls into the description later.

How shadow works

Basic shadow functionality

In order to produce correct shadow lines, line art will render a preliminary "light buffer", then all visible feature line segments ("light contours") from this buffer are cast onto triangles behind them, this would provide shadow lines as well as occlusion-adjusted light contours, which are then loaded with scene geometry/feature lines together into the "real" line art calculation where the final line result is generated. See the illustration below:

Note only visible segments from the "light camera" is projected onto surfaces behind them. This naturally prevents light contour that's inside shaow region to show, and it also combines the cast shadow into a continuous shape, which is a lot of times preferred in an artistic sense.

Why not ray trace?

This is because ray trace along one specific edge only has finite amount of sampling points, if a feature is smaller than the sampling interval length, the result can become ambiguous. Shadow reprojection can take advantage of the very robust vector-based occlusion algorithm that line art developed, which is also quite fast.

How does reproject work?

First, it ensures that it's only casting visible segments from the "light buffer", which means line art would run for the first time as if the selected light source is the camera, and finish occlusion computation (only for contour lines, obviously) before it can generate any shadow lines.

When to cast shadows, LineArt would put all those segments at infinite distance from the "light camera" (internally it's put at 1e30), then, for every triangle that's in front of it, LineArt find the nearest projection onto its surface. Eventually, every single segment would be "brought forth" by the algorithm to the closest surface possible (which is guaranteed to still be behind the original light contour, or otherwise that light contour won't be visible in the light buffer). This way, we could accurately project one edge onto multiple triangles, even when receiving triangles are intersecting. See illustration below for a more graphic representation:

Finally, we get everything projected, so each original edge has become one or more segments that are (likely) discontinuous. For each of those segments, LineArt generate one actual LineartEdge, and set reference to the receiving triangle of that specific segment and the original edge (for easier identifying in later occlusion stage).

Then, LineArt runs again, loads the same scene together with those generated "shadow edges" to generate the actual camera result.

Note: At the moment, LineArt only supports one light source, and it needs to be directional (nothing "behind" the light is computed, shadow will only cast on -Z direction from the light reference object). That's due to the limitation of "light camera", which understandably, nothing behind the camera is computed. So when using a point light, it will only lit up one side of it, which may seem counter intuitive. So my suggestion would be to avoid using line art shadow functionality when you have a scene which requires prominent shadow representation "around" an light source. For sun type, it should work correctly as long as your scene is properly lit.

Select lines only from lit or dark areas.

Because now we support shadow line generation, it's reasonable for artists to want to "select lines that comes from dark or lit region". This requires LineArt to reproject for yet another time, after main camera results are generated, to put camera-space feature lines into the "light buffer", do another occlusion query, then finally it will be able to determine which edge is or is not illuminated, thus enable the user to select edges from only within those areas. Of course, if LineArt modifier isn't configured to distinguish lit and shaded region, this reprojection stage is omitted.

Note: Contour and light contour line would be selected as "lit" lines when viewed from the lit face, otherwise it will be "shaded". This strategy allows easier calculation of enclosed shapes when camera is against light direction. Cast shadow would always be selected no matter "lit" or "shaded". ~~At the moment, light contour and cast shadow lines are treated as "illuminated". (They sit at the border of lit/shaded regions, what better idea would you have...~~)

Enclosed Shapes

When artist draw or paint, composition-wise it's very important to maintian a good shape design, a lot of times it requires combination of light and shadow shapes into a continous shape. Note that the basic shadow functionality naturally combines "cast shadow" shapes, but lit/dark shapes are still separated/discontinued in the following conditions:

"illuminated contours" inside "lit areas" break the shape.
Lack of separation line between lit and dark shapes.

Through the very same reprojection algorithm in "lit/shaded" selection, Enclosed Shapes functionality can determine those situations, and erase/show those lines respectively. (if a reprojected lit edge lands on a lit region, then it satisfies condition 1, if it's a shaded-edge, it satisfies condition 2). Once this process is done, you will have a solid line drawing that strictly draws light/shadow shapes, which is of very good artistic use. See illustration below for before/after effect (Both selected only "lit" region):

How Silhouette works

It may seem not related to shadow support, yet through the very same reprojection algorithm used to determine lit/shaded regions, LineArt is able to compute silhouette lines of groups of objects (up to 256 groups) very conveniently. A contour line becomes a silhouette only when (reprojected from camera) its reprojection shoots to infinity, or lands on the mesh that has a different silhouette_group number as the contour line, otherwise, it's a inner contour, or "anti-silhouette". This also gives an additional benefit of automatically separating silhouette groups when objects inside them intersect with each other, which is not very easy to effectively process in a ray-traced/filtering based algorithm. In the illustration below, the cigarette has a different silhouette group than the rest of the scene. Although it intersect with the monkey mesh, silhouette of both groups (black thick lines) are being calculated correctly without ordering/splitting the geometry etc.

Notes:

This is a (more or less?) comprehensive diagram showing how different stages are executed with accompanying function calls:

To achieve full functionality of shadow and silhouette, LineArt used two additional projections to calculate necessary information. Necessary stages will be automatically enabled when needed, and disabled by default when such data is not required.

(Other details to come later :P)

On `obindex` and edge/triangle matching

Line art needs to compute visibility for both view camera and light camera, and due to clipping and visibility rejection, geometries loaded in one view would not necessarily (or almost never) be the same as the other view, this brings a problem of needing to

Match edges and triangles from two views.
Record which triangle a projected segment landed on to, and get that information in both views.

In order for us to do this without using too much memory to store address references. LineArt approaches this problem by:

Assigning unique object index integer to each one of the object regardless of whether it's visibility rejected, and because the iteration order of loading the object is the same for both views, it's guaranteed that the index would match.
triangle->target_reference is assigned ((obindex << 20) | (triangle_index_in_object & 0xfffff)), which gives a unique edge identifier for both views.
shadow_segment->target_reference is assigned to the triangle that this shadow segment landed on, which is then assigned to edge->target_reference when loading projected shadow into view camera for computing occlusion. And during occlusion, if edge->target_reference == triangle->target_reference, then we know this edge is directly on the surface of the triangle thus we don't wast time dealing with floating point precision for identifying whether the triangle occludes that edge.
edge->from_shadow is assigned differently based on edge type:
- It's assigned to ((obindex<<20|v1->index)<<32) | (obindex<<20|v2->index) for mesh feature lines, so if we have any two edges that have the same from_shadow value, we know that they are the same edge from two views.
- It's assigned to (e->t1->target_reference<<32) | (e->t2->target_reference) for intersection lines, for the same purpose. (We only do this for intersection lines because they don't technically belong to any singular objects and they do not have indexed vertices associated with them)
Later, after projection and everything is done, LineArt finds matching edges from light camera and camera view edges, then registers shadow/silhouette cuts based on light camera occlusion results.

What if some edges in view camera falls outside the coverage of light camera?

In that case those edges doesn't have matching edges, thus LineArt couldn't get occlusion info for them. Currently neither "lit/shaded" info is assigned to them, meaning you can't select those edges by choosing either one. An option might be added to "treat those regions as if they were lit or shaded". This is very much similar to realtime graphics lighting where they only have a certain coverage, outside that you get no shadow. Suggestions are welcomed.

I decide to put the design notes in this separate task for the moment, so this way I can explain things much better, it will be a complement to the shadow patch (yet to come) information. I will add the corresponding function calls into the description later. ## How shadow works ### Basic shadow functionality In order to produce correct shadow lines, line art will render a preliminary "light buffer", then all visible feature line segments ("light contours") from this buffer are cast onto triangles behind them, this would provide shadow lines as well as occlusion-adjusted light contours, which are then loaded with scene geometry/feature lines together into the "real" line art calculation where the final line result is generated. See the illustration below: ![图片.png](https://archive.blender.org/developer/F13123135/图片.png) Note only visible segments from the "light camera" is projected onto surfaces behind them. This naturally prevents light contour that's inside shaow region to show, and it also combines the cast shadow into a continuous shape, which is a lot of times preferred in an artistic sense. **Why not ray trace?** This is because ray trace along one specific edge only has finite amount of sampling points, if a feature is smaller than the sampling interval length, the result can become ambiguous. Shadow reprojection can take advantage of the very robust vector-based occlusion algorithm that line art developed, which is also quite fast. **How does reproject work?** First, it ensures that it's only casting **visible segments** from the "light buffer", which means line art would run for the first time as if the selected light source is the camera, and finish occlusion computation (only for contour lines, obviously) before it can generate any shadow lines. When to cast shadows, LineArt would put all those segments at infinite distance from the "light camera" (internally it's put at `1e30`), then, for every triangle that's in front of it, LineArt find the nearest projection onto its surface. Eventually, every single segment would be "brought forth" by the algorithm to the closest surface possible (which is guaranteed to still be *behind* the original light contour, or otherwise that light contour won't be visible in the light buffer). This way, we could accurately project one edge onto multiple triangles, even when receiving triangles are intersecting. See illustration below for a more graphic representation: ![图片.png](https://archive.blender.org/developer/F13123089/图片.png) ![图片.png](https://archive.blender.org/developer/F13123104/图片.png) Finally, we get everything projected, so each original edge has become one or more segments that are (likely) discontinuous. For each of those segments, LineArt generate one actual `LineartEdge`, and set reference to the receiving triangle of that specific segment and the original edge (for easier identifying in later occlusion stage). Then, LineArt runs again, loads the same scene together with those generated "shadow edges" to generate the actual camera result. **Note**: At the moment, LineArt only supports one light source, and it needs to be directional (nothing "behind" the light is computed, shadow will only cast on `-Z` direction from the light reference object). That's due to the limitation of "light camera", which understandably, nothing behind the camera is computed. So when using a point light, it will only lit up one side of it, which may seem counter intuitive. So my suggestion would be to avoid using line art shadow functionality when you have a scene which requires prominent shadow representation "around" an light source. For sun type, it should work correctly as long as your scene is properly lit. ### Select lines only from lit or dark areas. Because now we support shadow line generation, it's reasonable for artists to want to "select lines that comes from dark or lit region". This requires LineArt to reproject for yet another time, after main camera results are generated, to put camera-space feature lines into the "light buffer", do another occlusion query, then finally it will be able to determine which edge is or is not illuminated, thus enable the user to select edges from only within those areas. Of course, if LineArt modifier isn't configured to distinguish lit and shaded region, this reprojection stage is omitted. **Note**: Contour and light contour line would be selected as "lit" lines when viewed from the lit face, otherwise it will be "shaded". This strategy allows easier calculation of enclosed shapes when camera is against light direction. Cast shadow would always be selected no matter "lit" or "shaded". ~~At the moment, light contour and cast shadow lines are treated as "illuminated". (They sit at the border of lit/shaded regions, what better idea would you have...~~) ### Enclosed Shapes When artist draw or paint, composition-wise it's very important to maintian a good shape design, a lot of times it requires combination of light and shadow shapes into a continous shape. Note that the basic shadow functionality naturally combines "cast shadow" shapes, but lit/dark shapes are still separated/discontinued in the following conditions: 1. "illuminated contours" inside "lit areas" break the shape. 2. Lack of separation line between lit and dark shapes. Through the very same reprojection algorithm in "lit/shaded" selection, Enclosed Shapes functionality can determine those situations, and erase/show those lines respectively. (if a reprojected lit edge lands on a lit region, then it satisfies condition 1, if it's a shaded-edge, it satisfies condition 2). Once this process is done, you will have a solid line drawing that strictly draws light/shadow shapes, which is of very good artistic use. See illustration below for before/after effect (Both selected only "lit" region): ![图片.png](https://archive.blender.org/developer/F13123188/图片.png) ![图片.png](https://archive.blender.org/developer/F13123190/图片.png) ## How Silhouette works It may seem not related to shadow support, yet through the very same reprojection algorithm used to determine lit/shaded regions, LineArt is able to compute silhouette lines of groups of objects (up to 256 groups) very conveniently. A contour line becomes a silhouette only when (reprojected from camera) its reprojection shoots to infinity, or lands on the mesh that has a different `silhouette_group` number as the contour line, otherwise, it's a inner contour, or "anti-silhouette". This also gives an additional benefit of automatically separating silhouette groups when objects inside them intersect with each other, which is not very easy to effectively process in a ray-traced/filtering based algorithm. In the illustration below, the cigarette has a different silhouette group than the rest of the scene. Although it intersect with the monkey mesh, silhouette of both groups (black thick lines) are being calculated correctly without ordering/splitting the geometry etc. ![图片.png](https://archive.blender.org/developer/F13123209/图片.png) ## Notes: This is a (more or less?) comprehensive diagram showing how different stages are executed with accompanying function calls: ![图片.png](https://archive.blender.org/developer/F13123967/图片.png) To achieve full functionality of shadow and silhouette, LineArt used two additional projections to calculate necessary information. Necessary stages will be automatically enabled when needed, and disabled by default when such data is not required. (Other details to come later :P) ### On `obindex` and edge/triangle matching Line art needs to compute visibility for both view camera and light camera, and due to clipping and visibility rejection, geometries loaded in one view would not necessarily (or almost never) be the same as the other view, this brings a problem of needing to 1. Match edges and triangles from two views. 2. Record which triangle a projected segment landed on to, and get that information in both views. In order for us to do this without using too much memory to store address references. LineArt approaches this problem by: - Assigning unique object index integer to each one of the object regardless of whether it's visibility rejected, and because the iteration order of loading the object is the same for both views, it's guaranteed that the index would match. - `triangle->target_reference` is assigned `((obindex << 20) | (triangle_index_in_object & 0xfffff))`, which gives a unique edge identifier for both views. - `shadow_segment->target_reference` is assigned to the triangle that this shadow segment landed on, which is then assigned to `edge->target_reference` when loading projected shadow into view camera for computing occlusion. And during occlusion, if `edge->target_reference == triangle->target_reference`, then we know this edge is directly on the surface of the triangle thus we don't wast time dealing with floating point precision for identifying whether the triangle occludes that edge. - `edge->from_shadow` is assigned differently based on edge type: - It's assigned to `((obindex<<20|v1->index)<<32) | (obindex<<20|v2->index)` for mesh feature lines, so if we have any two edges that have the same `from_shadow` value, we know that they are the same edge from two views. - It's assigned to `(e->t1->target_reference<<32) | (e->t2->target_reference)` for intersection lines, for the same purpose. (We only do this for intersection lines because they don't technically belong to any singular objects and they do not have indexed vertices associated with them) - Later, after projection and everything is done, LineArt finds matching edges from light camera and camera view edges, then registers shadow/silhouette cuts based on light camera occlusion results. **What if some edges in view camera falls outside the coverage of light camera?** In that case those edges doesn't have matching edges, thus LineArt couldn't get occlusion info for them. Currently neither "lit/shaded" info is assigned to them, meaning you can't select those edges by choosing either one. An option might be added to "treat those regions as if they were lit or shaded". This is very much similar to realtime graphics lighting where they only have a certain coverage, outside that you get no shadow. Suggestions are welcomed.

YimingWu self-assigned this 2022-05-30 15:55:12 +02:00

YimingWu commented

2022-05-30 15:55:12 +02:00

Added subscribers: @ChengduLittleA, @Yuro, @Hologram, @TinyNick, @fclem, @mendio, @okuma_10, @hzuika, @JSM, @frogstomp-4, @Sergey, @ZedDB, @Garek, @brecht, @bunny, @GeorgiaPacific, @antoniov

YimingWu commented

2022-05-30 16:44:59 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

blender-admin commented

2022-06-29 16:54:29 +02:00

This issue was referenced by 6dd8ceef2a

This issue was referenced by 6dd8ceef2a21f64cbb61a96560c50c162f9dae39

Robert S commented

2022-12-09 10:46:24 +01:00

Added subscriber: @RobertS

Robert S commented

2022-12-09 10:46:24 +01:00

This comment was removed by @RobertS

*This comment was removed by @RobertS*

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

LineArt shadow functionality design notes #98498

How shadow works

Basic shadow functionality

Select lines only from lit or dark areas.

Enclosed Shapes

How Silhouette works

Notes:

On obindex and edge/triangle matching

On `obindex` and edge/triangle matching