Realtime mode for line art (And more!) #89134

Open
opened 2021-06-14 10:52:26 +02:00 by YimingWu · 14 comments
Member

Currently line art is still slow to a point where a slightly complex scene will not be previewed in real-time. This is a problem needing to be addressed.

Because at the moment the line art modifier is a vector calculation, it's not as fast as GPU shaders, and works in a different way as well, which means there will be a lot of differences as of how the line is shaded and what kind of style can we apply.

There are some existing way of showing line art in real-time. The outline and mesh boundary "look" in the viewport is already a good example of real-time line art. Other method can be implemented as well for a real-time line art that "looks good enough", such as:

  • The GPU mode code in the soc 2018, which is from this paper: https://gfx.cs.princeton.edu/pubs/Cole_2009_FHL/cole_2009_fhl.pdf Robust, needs adjacent geometry info, pixel-level depth visibility test, need chaining for stylization.
  • Classic "inverted hull" approach, but this can potentially be implemented into a render engine automatic operation (just like when you plug in a displacement node). This method looks natural for organic shapes, but lack of stroke stylization.
  • Various ways to get real-time intersections, which can include involving CUDA and some newer APIs (4.x+?) in OpenGL that allows random access of lists (Not widely supported by older drivers). The most efficient way of doing intersections can be using inverted hull method and with some blending tricks in the depth buffers.
  • Image filtering and tracing, this gives flexibility with styling but as the soc 2018 code shows the performance is limited by the bandwidth of client/gpu data transfer, making it not very much desired in the first place.

Current line art code is getting more and more parallel, so it may eventually run on CUDA. All the "real-time" tricks we did up there won't be significantly faster if there's a large scene either.

Please give your thoughts on this :)


More stuff?

Because the workings of line art directly correlates to how people draw, this would naturally leads to shading and other advanced way of expressing geometry and shape forms. This could happen in current line art architecture, but could also be another module on top that get those results that are hard for conventional render engines to handle. With possible help from geometry nodes, some work can be done, we'll see.

Currently line art is still slow to a point where a slightly complex scene will not be previewed in real-time. This is a problem needing to be addressed. Because at the moment the line art modifier is a vector calculation, it's not as fast as GPU shaders, and works in a different way as well, which means there will be a lot of differences as of how the line is shaded and what kind of style can we apply. There are some existing way of showing line art in real-time. The outline and mesh boundary "look" in the viewport is already a good example of real-time line art. Other method can be implemented as well for a real-time line art that "looks good enough", such as: - The GPU mode code in the soc 2018, which is from this paper: https://gfx.cs.princeton.edu/pubs/Cole_2009_FHL/cole_2009_fhl.pdf Robust, needs adjacent geometry info, pixel-level depth visibility test, need chaining for stylization. - Classic "inverted hull" approach, but this can potentially be implemented into a render engine automatic operation (just like when you plug in a displacement node). This method looks natural for organic shapes, but lack of stroke stylization. - Various ways to get real-time intersections, which can include involving CUDA and some newer APIs (4.x+?) in OpenGL that allows random access of lists (Not widely supported by older drivers). The most efficient way of doing intersections can be using inverted hull method and with some blending tricks in the depth buffers. - Image filtering and tracing, this gives flexibility with styling but as the soc 2018 code shows the performance is limited by the bandwidth of client/gpu data transfer, making it not very much desired in the first place. Current line art code is getting more and more parallel, so it may eventually run on CUDA. All the "real-time" tricks we did up there won't be significantly faster if there's a large scene either. Please give your thoughts on this :) ------ **More stuff?** Because the workings of line art directly correlates to how people draw, this would naturally leads to shading and other advanced way of expressing geometry and shape forms. This could happen in current line art architecture, but could also be another module on top that get those results that are hard for conventional render engines to handle. With possible help from geometry nodes, some work can be done, we'll see.
YimingWu self-assigned this 2021-06-14 10:52:27 +02:00
Author
Member

Added subscribers: @ChengduLittleA, @bunny, @GeorgiaPacific, @antoniov

Added subscribers: @ChengduLittleA, @bunny, @GeorgiaPacific, @antoniov
Added subscribers: @fclem, @mendio, @pepe-school-land, @filedescriptor

I agree that LineArt should be faster, but there are two key points to consider:

a) LineArt is a modifier, so everything is based on creating strokes that can be manipulated by other modifier. We must not generate only "visible" GPU data or the modifier stack will not work.
b) Creating a "special" drawing engine code for LineArt is not a good idea. In fact, there is some work in progress to integrate GPencil with Eevee ... creating a LIneArt engine goes in the opposite direction.

I would like to hear @fclem opinion about GPU drawing and shaders.

Any other step to make it faster is perfect, but we have been working a lot to integrate GPencil in Blender to begin to separate it again. We cannot consider GPencil an "island" in Blender.

I agree that LineArt should be faster, but there are two key points to consider: a) LineArt is a modifier, so everything is based on creating strokes that can be manipulated by other modifier. We must not generate only "visible" GPU data or the modifier stack will not work. b) Creating a "special" drawing engine code for LineArt is not a good idea. In fact, there is some work in progress to integrate GPencil with Eevee ... creating a LIneArt engine goes in the opposite direction. I would like to hear @fclem opinion about GPU drawing and shaders. Any other step to make it faster is perfect, but we have been working a lot to integrate GPencil in Blender to begin to separate it again. We cannot consider GPencil an "island" in Blender.

Added subscriber: @frogstomp-4

Added subscriber: @frogstomp-4

I need to back @antoniov on this. But here are my two cents:

  • The fact that we're now getting vector based lineart - one can simply not do any better that this. This is revolution in the making and makes blender unique among competitors.

  • There are competing forces for reinventing the wheel going on. For example we have MALT shader already, that needs to be manually downloaded. But it does exactly what it is proposed - but I have to say it also becomes slow quickly with larger scenes.

I have noticed that in blender community, there is general confusion on lineart approaches. And although it has strong reputation for general NPR - lineart is still not a selling point.

So my idea would be getting Malt officially integrated rendering engine and making lineart template where one could switch between malt lines and grease pencil lineart. There simply is no point to reinvent this.

Malt example:
image.png
Lineart example:
image.png

I need to back @antoniov on this. But here are my two cents: - The fact that we're now getting vector based lineart - one can simply not do any better that this. This is revolution in the making and makes blender unique among competitors. - There are competing forces for reinventing the wheel going on. For example we have MALT shader already, that needs to be manually downloaded. But it does exactly what it is proposed - but I have to say it also becomes slow quickly with larger scenes. I have noticed that in blender community, there is general confusion on lineart approaches. And although it has strong reputation for general NPR - lineart is still not a selling point. So my idea would be getting Malt officially integrated rendering engine and making lineart template where one could switch between malt lines and grease pencil lineart. There simply is no point to reinvent this. Malt example: ![image.png](https://archive.blender.org/developer/F10170735/image.png) Lineart example: ![image.png](https://archive.blender.org/developer/F10170736/image.png)
Author
Member

Added subscriber: @ZedDB

Added subscriber: @ZedDB
Author
Member

b) Creating a "special" drawing engine code for LineArt is not a good idea. In fact, there is some work in progress to integrate GPencil with Eevee ... creating a LIneArt engine goes in the opposite direction.

I think it can be a simple overlay? Even when eevee takes care of gp there will still be overlays drawing on top, like selections and so on. This "preview" line should be fine in this sense.

@ZedDB suggested in the chat that CUDA line art is the best we could expect for current architecture. This is because we can have "near real time" performance and not lose any styling ability.

I have noticed that in blender community, there is general confusion on lineart approaches. And although it has strong reputation for general NPR - lineart is still not a selling point.

Maybe with future shadow shadow lines/reflections/camera distortion stuff in here, things can change a bit, but I'm not sure. These have always been line renderer's problem.

> b) Creating a "special" drawing engine code for LineArt is not a good idea. In fact, there is some work in progress to integrate GPencil with Eevee ... creating a LIneArt engine goes in the opposite direction. I think it can be a simple overlay? Even when eevee takes care of gp there will still be overlays drawing on top, like selections and so on. This "preview" line should be fine in this sense. @ZedDB suggested in the chat that CUDA line art is the best we could expect for current architecture. This is because we can have "near real time" performance and not lose any styling ability. > I have noticed that in blender community, there is general confusion on lineart approaches. And although it has strong reputation for general NPR - lineart is still not a selling point. Maybe with future shadow shadow lines/reflections/camera distortion stuff in here, things can change a bit, but I'm not sure. These have always been line renderer's problem.
YimingWu changed title from Realtime mode for line art to Realtime mode for line art (And more!) 2021-06-15 16:54:16 +02:00

Added subscriber: @Grady

Added subscriber: @Grady

Perhaps this should be treated as a special case 'effect', similar to the 'bloom' effect for Eevee? Perhaps a render setting that can be enabled that adds simple real time line art overlay to a scene for when someone wants a simple cheap effect and doesn't so much care about calculating a complex editable grease pencil outline.

Perhaps this should be treated as a special case 'effect', similar to the 'bloom' effect for Eevee? Perhaps a render setting that can be enabled that adds simple real time line art overlay to a scene for when someone wants a simple cheap effect and doesn't so much care about calculating a complex editable grease pencil outline.

Added subscriber: @tsatse

Added subscriber: @tsatse

Removed subscriber: @tsatse

Removed subscriber: @tsatse
Author
Member

In #89134#1177241, @Grady wrote:
Perhaps this should be treated as a special case 'effect', similar to the 'bloom' effect for Eevee? Perhaps a render setting that can be enabled that adds simple real time line art overlay to a scene for when someone wants a simple cheap effect and doesn't so much care about calculating a complex editable grease pencil outline.

Yes, let's keep this option open. I'm not entirely sure how future node system is gonna benefit the rendering process, and how real-time that can be, so yeah, we could use some eye-candy if nothing proves to be really-real-time :D

> In #89134#1177241, @Grady wrote: > Perhaps this should be treated as a special case 'effect', similar to the 'bloom' effect for Eevee? Perhaps a render setting that can be enabled that adds simple real time line art overlay to a scene for when someone wants a simple cheap effect and doesn't so much care about calculating a complex editable grease pencil outline. Yes, let's keep this option open. I'm not entirely sure how future node system is gonna benefit the rendering process, and how real-time that can be, so yeah, we could use some eye-candy if nothing proves to be really-real-time :D

Added subscriber: @brecht

Added subscriber: @brecht

Detaching task from a specific Blender release. In general this should only be used when the feature is being worked on and targeted for a specific release, rather than a general module task.

Detaching task from a specific Blender release. In general this should only be used when the feature is being worked on and targeted for a specific release, rather than a general module task.
Philipp Oeser removed the
Interest
Grease Pencil
label 2023-02-09 15:19:26 +01:00
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset System
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Asset Browser Project
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
6 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#89134
No description provided.