Realtime mode for line art (And more!) #89134

New Issue

YimingWu · 2021-06-14T10:52:26+02:00

YimingWu commented

2021-06-14 10:52:26 +02:00

Currently line art is still slow to a point where a slightly complex scene will not be previewed in real-time. This is a problem needing to be addressed.

Because at the moment the line art modifier is a vector calculation, it's not as fast as GPU shaders, and works in a different way as well, which means there will be a lot of differences as of how the line is shaded and what kind of style can we apply.

There are some existing way of showing line art in real-time. The outline and mesh boundary "look" in the viewport is already a good example of real-time line art. Other method can be implemented as well for a real-time line art that "looks good enough", such as:

The GPU mode code in the soc 2018, which is from this paper: https://gfx.cs.princeton.edu/pubs/Cole_2009_FHL/cole_2009_fhl.pdf Robust, needs adjacent geometry info, pixel-level depth visibility test, need chaining for stylization.
Classic "inverted hull" approach, but this can potentially be implemented into a render engine automatic operation (just like when you plug in a displacement node). This method looks natural for organic shapes, but lack of stroke stylization.
Various ways to get real-time intersections, which can include involving CUDA and some newer APIs (4.x+?) in OpenGL that allows random access of lists (Not widely supported by older drivers). The most efficient way of doing intersections can be using inverted hull method and with some blending tricks in the depth buffers.
Image filtering and tracing, this gives flexibility with styling but as the soc 2018 code shows the performance is limited by the bandwidth of client/gpu data transfer, making it not very much desired in the first place.

Current line art code is getting more and more parallel, so it may eventually run on CUDA. All the "real-time" tricks we did up there won't be significantly faster if there's a large scene either.

Please give your thoughts on this :)

More stuff?

Because the workings of line art directly correlates to how people draw, this would naturally leads to shading and other advanced way of expressing geometry and shape forms. This could happen in current line art architecture, but could also be another module on top that get those results that are hard for conventional render engines to handle. With possible help from geometry nodes, some work can be done, we'll see.

Currently line art is still slow to a point where a slightly complex scene will not be previewed in real-time. This is a problem needing to be addressed. Because at the moment the line art modifier is a vector calculation, it's not as fast as GPU shaders, and works in a different way as well, which means there will be a lot of differences as of how the line is shaded and what kind of style can we apply. There are some existing way of showing line art in real-time. The outline and mesh boundary "look" in the viewport is already a good example of real-time line art. Other method can be implemented as well for a real-time line art that "looks good enough", such as: - The GPU mode code in the soc 2018, which is from this paper: https://gfx.cs.princeton.edu/pubs/Cole_2009_FHL/cole_2009_fhl.pdf Robust, needs adjacent geometry info, pixel-level depth visibility test, need chaining for stylization. - Classic "inverted hull" approach, but this can potentially be implemented into a render engine automatic operation (just like when you plug in a displacement node). This method looks natural for organic shapes, but lack of stroke stylization. - Various ways to get real-time intersections, which can include involving CUDA and some newer APIs (4.x+?) in OpenGL that allows random access of lists (Not widely supported by older drivers). The most efficient way of doing intersections can be using inverted hull method and with some blending tricks in the depth buffers. - Image filtering and tracing, this gives flexibility with styling but as the soc 2018 code shows the performance is limited by the bandwidth of client/gpu data transfer, making it not very much desired in the first place. Current line art code is getting more and more parallel, so it may eventually run on CUDA. All the "real-time" tricks we did up there won't be significantly faster if there's a large scene either. Please give your thoughts on this :) ------ **More stuff?** Because the workings of line art directly correlates to how people draw, this would naturally leads to shading and other advanced way of expressing geometry and shape forms. This could happen in current line art architecture, but could also be another module on top that get those results that are hard for conventional render engines to handle. With possible help from geometry nodes, some work can be done, we'll see.

YimingWu self-assigned this 2021-06-14 10:52:27 +02:00

YimingWu commented

2021-06-14 10:52:27 +02:00

Added subscribers: @ChengduLittleA, @bunny, @GeorgiaPacific, @antoniov

Antonio Vazquez commented

2021-06-14 11:08:23 +02:00

Added subscribers: @fclem, @mendio, @pepe-school-land, @filedescriptor

Antonio Vazquez commented

2021-06-14 11:08:23 +02:00

I agree that LineArt should be faster, but there are two key points to consider:

a) LineArt is a modifier, so everything is based on creating strokes that can be manipulated by other modifier. We must not generate only "visible" GPU data or the modifier stack will not work.
b) Creating a "special" drawing engine code for LineArt is not a good idea. In fact, there is some work in progress to integrate GPencil with Eevee ... creating a LIneArt engine goes in the opposite direction.

I would like to hear @fclem opinion about GPU drawing and shaders.

Any other step to make it faster is perfect, but we have been working a lot to integrate GPencil in Blender to begin to separate it again. We cannot consider GPencil an "island" in Blender.

I agree that LineArt should be faster, but there are two key points to consider: a) LineArt is a modifier, so everything is based on creating strokes that can be manipulated by other modifier. We must not generate only "visible" GPU data or the modifier stack will not work. b) Creating a "special" drawing engine code for LineArt is not a good idea. In fact, there is some work in progress to integrate GPencil with Eevee ... creating a LIneArt engine goes in the opposite direction. I would like to hear @fclem opinion about GPU drawing and shaders. Any other step to make it faster is perfect, but we have been working a lot to integrate GPencil in Blender to begin to separate it again. We cannot consider GPencil an "island" in Blender.

Aleš Jelovčan commented

2021-06-14 11:24:53 +02:00

Added subscriber: @frogstomp-4

Aleš Jelovčan commented

2021-06-14 11:24:53 +02:00

I need to back @antoniov on this. But here are my two cents:

The fact that we're now getting vector based lineart - one can simply not do any better that this. This is revolution in the making and makes blender unique among competitors.
There are competing forces for reinventing the wheel going on. For example we have MALT shader already, that needs to be manually downloaded. But it does exactly what it is proposed - but I have to say it also becomes slow quickly with larger scenes.

I have noticed that in blender community, there is general confusion on lineart approaches. And although it has strong reputation for general NPR - lineart is still not a selling point.

So my idea would be getting Malt officially integrated rendering engine and making lineart template where one could switch between malt lines and grease pencil lineart. There simply is no point to reinvent this.

Malt example:

Lineart example:

I need to back @antoniov on this. But here are my two cents: - The fact that we're now getting vector based lineart - one can simply not do any better that this. This is revolution in the making and makes blender unique among competitors. - There are competing forces for reinventing the wheel going on. For example we have MALT shader already, that needs to be manually downloaded. But it does exactly what it is proposed - but I have to say it also becomes slow quickly with larger scenes. I have noticed that in blender community, there is general confusion on lineart approaches. And although it has strong reputation for general NPR - lineart is still not a selling point. So my idea would be getting Malt officially integrated rendering engine and making lineart template where one could switch between malt lines and grease pencil lineart. There simply is no point to reinvent this. Malt example: ![image.png](https://archive.blender.org/developer/F10170735/image.png) Lineart example: ![image.png](https://archive.blender.org/developer/F10170736/image.png)

YimingWu commented

2021-06-14 11:43:23 +02:00

Added subscriber: @ZedDB

YimingWu commented

2021-06-14 11:43:23 +02:00

b) Creating a "special" drawing engine code for LineArt is not a good idea. In fact, there is some work in progress to integrate GPencil with Eevee ... creating a LIneArt engine goes in the opposite direction.

I think it can be a simple overlay? Even when eevee takes care of gp there will still be overlays drawing on top, like selections and so on. This "preview" line should be fine in this sense.

@ZedDB suggested in the chat that CUDA line art is the best we could expect for current architecture. This is because we can have "near real time" performance and not lose any styling ability.

I have noticed that in blender community, there is general confusion on lineart approaches. And although it has strong reputation for general NPR - lineart is still not a selling point.

Maybe with future shadow shadow lines/reflections/camera distortion stuff in here, things can change a bit, but I'm not sure. These have always been line renderer's problem.

> b) Creating a "special" drawing engine code for LineArt is not a good idea. In fact, there is some work in progress to integrate GPencil with Eevee ... creating a LIneArt engine goes in the opposite direction. I think it can be a simple overlay? Even when eevee takes care of gp there will still be overlays drawing on top, like selections and so on. This "preview" line should be fine in this sense. @ZedDB suggested in the chat that CUDA line art is the best we could expect for current architecture. This is because we can have "near real time" performance and not lose any styling ability. > I have noticed that in blender community, there is general confusion on lineart approaches. And although it has strong reputation for general NPR - lineart is still not a selling point. Maybe with future shadow shadow lines/reflections/camera distortion stuff in here, things can change a bit, but I'm not sure. These have always been line renderer's problem.

YimingWu changed title from ~~Realtime mode for line art~~ to Realtime mode for line art (And more!)

2021-06-15 16:54:16 +02:00

Mindinsomnia commented

2021-06-15 17:54:37 +02:00

Added subscriber: @Grady

Mindinsomnia commented

2021-06-15 17:54:37 +02:00

Perhaps this should be treated as a special case 'effect', similar to the 'bloom' effect for Eevee? Perhaps a render setting that can be enabled that adds simple real time line art overlay to a scene for when someone wants a simple cheap effect and doesn't so much care about calculating a complex editable grease pencil outline.

tsatse commented

2021-06-16 00:23:22 +02:00

Added subscriber: @tsatse

tsatse commented

2021-06-16 00:28:32 +02:00

Removed subscriber: @tsatse

YimingWu commented

2021-06-16 06:08:48 +02:00

In #89134#1177241, @Grady wrote:
Perhaps this should be treated as a special case 'effect', similar to the 'bloom' effect for Eevee? Perhaps a render setting that can be enabled that adds simple real time line art overlay to a scene for when someone wants a simple cheap effect and doesn't so much care about calculating a complex editable grease pencil outline.

Yes, let's keep this option open. I'm not entirely sure how future node system is gonna benefit the rendering process, and how real-time that can be, so yeah, we could use some eye-candy if nothing proves to be really-real-time :D

> In #89134#1177241, @Grady wrote: > Perhaps this should be treated as a special case 'effect', similar to the 'bloom' effect for Eevee? Perhaps a render setting that can be enabled that adds simple real time line art overlay to a scene for when someone wants a simple cheap effect and doesn't so much care about calculating a complex editable grease pencil outline. Yes, let's keep this option open. I'm not entirely sure how future node system is gonna benefit the rendering process, and how real-time that can be, so yeah, we could use some eye-candy if nothing proves to be really-real-time :D

Brecht Van Lommel commented

2021-11-24 12:45:10 +01:00

Added subscriber: @brecht

Brecht Van Lommel commented

2021-11-24 12:45:10 +01:00

Detaching task from a specific Blender release. In general this should only be used when the feature is being worked on and targeted for a specific release, rather than a general module task.

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Realtime mode for line art (And more!) #89134