LineArt shadow functionality design notes #98498

Open
opened 2022-05-30 15:55:12 +02:00 by YimingWu · 5 comments
Member

I decide to put the design notes in this separate task for the moment, so this way I can explain things much better, it will be a complement to the shadow patch (yet to come) information.

I will add the corresponding function calls into the description later.

How shadow works

Basic shadow functionality

In order to produce correct shadow lines, line art will render a preliminary "light buffer", then all visible feature line segments ("light contours") from this buffer are cast onto triangles behind them, this would provide shadow lines as well as occlusion-adjusted light contours, which are then loaded with scene geometry/feature lines together into the "real" line art calculation where the final line result is generated. See the illustration below:

图片.png

Note only visible segments from the "light camera" is projected onto surfaces behind them. This naturally prevents light contour that's inside shaow region to show, and it also combines the cast shadow into a continuous shape, which is a lot of times preferred in an artistic sense.

Why not ray trace?

This is because ray trace along one specific edge only has finite amount of sampling points, if a feature is smaller than the sampling interval length, the result can become ambiguous. Shadow reprojection can take advantage of the very robust vector-based occlusion algorithm that line art developed, which is also quite fast.

How does reproject work?

First, it ensures that it's only casting visible segments from the "light buffer", which means line art would run for the first time as if the selected light source is the camera, and finish occlusion computation (only for contour lines, obviously) before it can generate any shadow lines.

When to cast shadows, LineArt would put all those segments at infinite distance from the "light camera" (internally it's put at 1e30), then, for every triangle that's in front of it, LineArt find the nearest projection onto its surface. Eventually, every single segment would be "brought forth" by the algorithm to the closest surface possible (which is guaranteed to still be behind the original light contour, or otherwise that light contour won't be visible in the light buffer). This way, we could accurately project one edge onto multiple triangles, even when receiving triangles are intersecting. See illustration below for a more graphic representation:

图片.png

图片.png

Finally, we get everything projected, so each original edge has become one or more segments that are (likely) discontinuous. For each of those segments, LineArt generate one actual LineartEdge, and set reference to the receiving triangle of that specific segment and the original edge (for easier identifying in later occlusion stage).

Then, LineArt runs again, loads the same scene together with those generated "shadow edges" to generate the actual camera result.

Note: At the moment, LineArt only supports one light source, and it needs to be directional (nothing "behind" the light is computed, shadow will only cast on -Z direction from the light reference object). That's due to the limitation of "light camera", which understandably, nothing behind the camera is computed. So when using a point light, it will only lit up one side of it, which may seem counter intuitive. So my suggestion would be to avoid using line art shadow functionality when you have a scene which requires prominent shadow representation "around" an light source. For sun type, it should work correctly as long as your scene is properly lit.

Select lines only from lit or dark areas.

Because now we support shadow line generation, it's reasonable for artists to want to "select lines that comes from dark or lit region". This requires LineArt to reproject for yet another time, after main camera results are generated, to put camera-space feature lines into the "light buffer", do another occlusion query, then finally it will be able to determine which edge is or is not illuminated, thus enable the user to select edges from only within those areas. Of course, if LineArt modifier isn't configured to distinguish lit and shaded region, this reprojection stage is omitted.

Note: Contour and light contour line would be selected as "lit" lines when viewed from the lit face, otherwise it will be "shaded". This strategy allows easier calculation of enclosed shapes when camera is against light direction. Cast shadow would always be selected no matter "lit" or "shaded". At the moment, light contour and cast shadow lines are treated as "illuminated". (They sit at the border of lit/shaded regions, what better idea would you have...)

Enclosed Shapes

When artist draw or paint, composition-wise it's very important to maintian a good shape design, a lot of times it requires combination of light and shadow shapes into a continous shape. Note that the basic shadow functionality naturally combines "cast shadow" shapes, but lit/dark shapes are still separated/discontinued in the following conditions:

  1. "illuminated contours" inside "lit areas" break the shape.
  2. Lack of separation line between lit and dark shapes.

Through the very same reprojection algorithm in "lit/shaded" selection, Enclosed Shapes functionality can determine those situations, and erase/show those lines respectively. (if a reprojected lit edge lands on a lit region, then it satisfies condition 1, if it's a shaded-edge, it satisfies condition 2). Once this process is done, you will have a solid line drawing that strictly draws light/shadow shapes, which is of very good artistic use. See illustration below for before/after effect (Both selected only "lit" region):

图片.png
图片.png

How Silhouette works

It may seem not related to shadow support, yet through the very same reprojection algorithm used to determine lit/shaded regions, LineArt is able to compute silhouette lines of groups of objects (up to 256 groups) very conveniently. A contour line becomes a silhouette only when (reprojected from camera) its reprojection shoots to infinity, or lands on the mesh that has a different silhouette_group number as the contour line, otherwise, it's a inner contour, or "anti-silhouette". This also gives an additional benefit of automatically separating silhouette groups when objects inside them intersect with each other, which is not very easy to effectively process in a ray-traced/filtering based algorithm. In the illustration below, the cigarette has a different silhouette group than the rest of the scene. Although it intersect with the monkey mesh, silhouette of both groups (black thick lines) are being calculated correctly without ordering/splitting the geometry etc.

图片.png

Notes:

This is a (more or less?) comprehensive diagram showing how different stages are executed with accompanying function calls:

图片.png

To achieve full functionality of shadow and silhouette, LineArt used two additional projections to calculate necessary information. Necessary stages will be automatically enabled when needed, and disabled by default when such data is not required.

(Other details to come later :P)

On obindex and edge/triangle matching

Line art needs to compute visibility for both view camera and light camera, and due to clipping and visibility rejection, geometries loaded in one view would not necessarily (or almost never) be the same as the other view, this brings a problem of needing to

  1. Match edges and triangles from two views.
  2. Record which triangle a projected segment landed on to, and get that information in both views.

In order for us to do this without using too much memory to store address references. LineArt approaches this problem by:

  • Assigning unique object index integer to each one of the object regardless of whether it's visibility rejected, and because the iteration order of loading the object is the same for both views, it's guaranteed that the index would match.
  • triangle->target_reference is assigned ((obindex << 20) | (triangle_index_in_object & 0xfffff)), which gives a unique edge identifier for both views.
  • shadow_segment->target_reference is assigned to the triangle that this shadow segment landed on, which is then assigned to edge->target_reference when loading projected shadow into view camera for computing occlusion. And during occlusion, if edge->target_reference == triangle->target_reference, then we know this edge is directly on the surface of the triangle thus we don't wast time dealing with floating point precision for identifying whether the triangle occludes that edge.
  • edge->from_shadow is assigned differently based on edge type:
    • It's assigned to ((obindex<<20|v1->index)<<32) | (obindex<<20|v2->index) for mesh feature lines, so if we have any two edges that have the same from_shadow value, we know that they are the same edge from two views.
    • It's assigned to (e->t1->target_reference<<32) | (e->t2->target_reference) for intersection lines, for the same purpose. (We only do this for intersection lines because they don't technically belong to any singular objects and they do not have indexed vertices associated with them)
  • Later, after projection and everything is done, LineArt finds matching edges from light camera and camera view edges, then registers shadow/silhouette cuts based on light camera occlusion results.

What if some edges in view camera falls outside the coverage of light camera?

In that case those edges doesn't have matching edges, thus LineArt couldn't get occlusion info for them. Currently neither "lit/shaded" info is assigned to them, meaning you can't select those edges by choosing either one. An option might be added to "treat those regions as if they were lit or shaded". This is very much similar to realtime graphics lighting where they only have a certain coverage, outside that you get no shadow. Suggestions are welcomed.

I decide to put the design notes in this separate task for the moment, so this way I can explain things much better, it will be a complement to the shadow patch (yet to come) information. I will add the corresponding function calls into the description later. ## How shadow works ### Basic shadow functionality In order to produce correct shadow lines, line art will render a preliminary "light buffer", then all visible feature line segments ("light contours") from this buffer are cast onto triangles behind them, this would provide shadow lines as well as occlusion-adjusted light contours, which are then loaded with scene geometry/feature lines together into the "real" line art calculation where the final line result is generated. See the illustration below: ![图片.png](https://archive.blender.org/developer/F13123135/图片.png) Note only visible segments from the "light camera" is projected onto surfaces behind them. This naturally prevents light contour that's inside shaow region to show, and it also combines the cast shadow into a continuous shape, which is a lot of times preferred in an artistic sense. **Why not ray trace?** This is because ray trace along one specific edge only has finite amount of sampling points, if a feature is smaller than the sampling interval length, the result can become ambiguous. Shadow reprojection can take advantage of the very robust vector-based occlusion algorithm that line art developed, which is also quite fast. **How does reproject work?** First, it ensures that it's only casting **visible segments** from the "light buffer", which means line art would run for the first time as if the selected light source is the camera, and finish occlusion computation (only for contour lines, obviously) before it can generate any shadow lines. When to cast shadows, LineArt would put all those segments at infinite distance from the "light camera" (internally it's put at `1e30`), then, for every triangle that's in front of it, LineArt find the nearest projection onto its surface. Eventually, every single segment would be "brought forth" by the algorithm to the closest surface possible (which is guaranteed to still be *behind* the original light contour, or otherwise that light contour won't be visible in the light buffer). This way, we could accurately project one edge onto multiple triangles, even when receiving triangles are intersecting. See illustration below for a more graphic representation: ![图片.png](https://archive.blender.org/developer/F13123089/图片.png) ![图片.png](https://archive.blender.org/developer/F13123104/图片.png) Finally, we get everything projected, so each original edge has become one or more segments that are (likely) discontinuous. For each of those segments, LineArt generate one actual `LineartEdge`, and set reference to the receiving triangle of that specific segment and the original edge (for easier identifying in later occlusion stage). Then, LineArt runs again, loads the same scene together with those generated "shadow edges" to generate the actual camera result. **Note**: At the moment, LineArt only supports one light source, and it needs to be directional (nothing "behind" the light is computed, shadow will only cast on `-Z` direction from the light reference object). That's due to the limitation of "light camera", which understandably, nothing behind the camera is computed. So when using a point light, it will only lit up one side of it, which may seem counter intuitive. So my suggestion would be to avoid using line art shadow functionality when you have a scene which requires prominent shadow representation "around" an light source. For sun type, it should work correctly as long as your scene is properly lit. ### Select lines only from lit or dark areas. Because now we support shadow line generation, it's reasonable for artists to want to "select lines that comes from dark or lit region". This requires LineArt to reproject for yet another time, after main camera results are generated, to put camera-space feature lines into the "light buffer", do another occlusion query, then finally it will be able to determine which edge is or is not illuminated, thus enable the user to select edges from only within those areas. Of course, if LineArt modifier isn't configured to distinguish lit and shaded region, this reprojection stage is omitted. **Note**: Contour and light contour line would be selected as "lit" lines when viewed from the lit face, otherwise it will be "shaded". This strategy allows easier calculation of enclosed shapes when camera is against light direction. Cast shadow would always be selected no matter "lit" or "shaded". ~~At the moment, light contour and cast shadow lines are treated as "illuminated". (They sit at the border of lit/shaded regions, what better idea would you have...~~) ### Enclosed Shapes When artist draw or paint, composition-wise it's very important to maintian a good shape design, a lot of times it requires combination of light and shadow shapes into a continous shape. Note that the basic shadow functionality naturally combines "cast shadow" shapes, but lit/dark shapes are still separated/discontinued in the following conditions: 1. "illuminated contours" inside "lit areas" break the shape. 2. Lack of separation line between lit and dark shapes. Through the very same reprojection algorithm in "lit/shaded" selection, Enclosed Shapes functionality can determine those situations, and erase/show those lines respectively. (if a reprojected lit edge lands on a lit region, then it satisfies condition 1, if it's a shaded-edge, it satisfies condition 2). Once this process is done, you will have a solid line drawing that strictly draws light/shadow shapes, which is of very good artistic use. See illustration below for before/after effect (Both selected only "lit" region): ![图片.png](https://archive.blender.org/developer/F13123188/图片.png) ![图片.png](https://archive.blender.org/developer/F13123190/图片.png) ## How Silhouette works It may seem not related to shadow support, yet through the very same reprojection algorithm used to determine lit/shaded regions, LineArt is able to compute silhouette lines of groups of objects (up to 256 groups) very conveniently. A contour line becomes a silhouette only when (reprojected from camera) its reprojection shoots to infinity, or lands on the mesh that has a different `silhouette_group` number as the contour line, otherwise, it's a inner contour, or "anti-silhouette". This also gives an additional benefit of automatically separating silhouette groups when objects inside them intersect with each other, which is not very easy to effectively process in a ray-traced/filtering based algorithm. In the illustration below, the cigarette has a different silhouette group than the rest of the scene. Although it intersect with the monkey mesh, silhouette of both groups (black thick lines) are being calculated correctly without ordering/splitting the geometry etc. ![图片.png](https://archive.blender.org/developer/F13123209/图片.png) ## Notes: This is a (more or less?) comprehensive diagram showing how different stages are executed with accompanying function calls: ![图片.png](https://archive.blender.org/developer/F13123967/图片.png) To achieve full functionality of shadow and silhouette, LineArt used two additional projections to calculate necessary information. Necessary stages will be automatically enabled when needed, and disabled by default when such data is not required. (Other details to come later :P) ### On `obindex` and edge/triangle matching Line art needs to compute visibility for both view camera and light camera, and due to clipping and visibility rejection, geometries loaded in one view would not necessarily (or almost never) be the same as the other view, this brings a problem of needing to 1. Match edges and triangles from two views. 2. Record which triangle a projected segment landed on to, and get that information in both views. In order for us to do this without using too much memory to store address references. LineArt approaches this problem by: - Assigning unique object index integer to each one of the object regardless of whether it's visibility rejected, and because the iteration order of loading the object is the same for both views, it's guaranteed that the index would match. - `triangle->target_reference` is assigned `((obindex << 20) | (triangle_index_in_object & 0xfffff))`, which gives a unique edge identifier for both views. - `shadow_segment->target_reference` is assigned to the triangle that this shadow segment landed on, which is then assigned to `edge->target_reference` when loading projected shadow into view camera for computing occlusion. And during occlusion, if `edge->target_reference == triangle->target_reference`, then we know this edge is directly on the surface of the triangle thus we don't wast time dealing with floating point precision for identifying whether the triangle occludes that edge. - `edge->from_shadow` is assigned differently based on edge type: - It's assigned to `((obindex<<20|v1->index)<<32) | (obindex<<20|v2->index)` for mesh feature lines, so if we have any two edges that have the same `from_shadow` value, we know that they are the same edge from two views. - It's assigned to `(e->t1->target_reference<<32) | (e->t2->target_reference)` for intersection lines, for the same purpose. (We only do this for intersection lines because they don't technically belong to any singular objects and they do not have indexed vertices associated with them) - Later, after projection and everything is done, LineArt finds matching edges from light camera and camera view edges, then registers shadow/silhouette cuts based on light camera occlusion results. **What if some edges in view camera falls outside the coverage of light camera?** In that case those edges doesn't have matching edges, thus LineArt couldn't get occlusion info for them. Currently neither "lit/shaded" info is assigned to them, meaning you can't select those edges by choosing either one. An option might be added to "treat those regions as if they were lit or shaded". This is very much similar to realtime graphics lighting where they only have a certain coverage, outside that you get no shadow. Suggestions are welcomed.
YimingWu self-assigned this 2022-05-30 15:55:12 +02:00
Author
Member
Added subscribers: @ChengduLittleA, @Yuro, @Hologram, @TinyNick, @fclem, @mendio, @okuma_10, @hzuika, @JSM, @frogstomp-4, @Sergey, @ZedDB, @Garek, @brecht, @bunny, @GeorgiaPacific, @antoniov
Author
Member

Changed status from 'Needs Triage' to: 'Confirmed'

Changed status from 'Needs Triage' to: 'Confirmed'

This issue was referenced by 6dd8ceef2a

This issue was referenced by 6dd8ceef2a21f64cbb61a96560c50c162f9dae39

Added subscriber: @RobertS

Added subscriber: @RobertS

This comment was removed by @RobertS

*This comment was removed by @RobertS*
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset System
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Asset Browser Project
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#98498
No description provided.