OpenGL headless rendering #54638

New Issue

Clément Foucault · 2018-04-16T12:21:57+02:00

Clément Foucault commented

2018-04-16 12:21:57 +02:00

This depends on #54633 and should be tackled afterwards.

We should allow OpenGL rendering on headless system (servers without display setup).

Resources on the subject are few but it seems our best shot is to use EGL. We need to assess the support of EGL across our target platforms, and the opengl features it provides.

This depends on #54633 and should be tackled afterwards. We should allow OpenGL rendering on headless system (servers without display setup). Resources on the subject are few but it seems our best shot is to use EGL. We need to assess the support of EGL across our target platforms, and the opengl features it provides.

👍 1

Clément Foucault self-assigned this 2018-04-16 12:21:57 +02:00

Clément Foucault commented

2018-04-16 12:21:57 +02:00

Added subscriber: @fclem

Dalai Felinto changed title from ~~Opengl headless rendering~~ to OpenGL headless rendering

2018-04-16 12:25:07 +02:00

Clément Foucault commented

2018-05-03 12:12:26 +02:00

Added subscriber: @Sergey

Clément Foucault commented

2018-05-03 12:12:26 +02:00

So after trying a few things on a standalone test application, we manage to create a EGL context with GL 4.6 on an Nvidia + Linux + Proprietary drivers workstation.
But we did not succeed on a AMD + Linux + Opensource drivers, only giving us a GL 3.0 context at most.

So this is highly platform dependant and might need more work to get it working nicely.

There is already an EGL backend to GHOST but it's targeted at GLES and outdated.

So we can divide this into 2 tasks:

Create compatible GL > 3.3 context with EGL (with maximum platform compatibility).
Add offscreen context functions to EGL backends.

This is not a really hard task but it is not a priority for now. So postponing for now.

Here is the test application we used : P671

Thanks @Sergey for helping me on this one.

So after trying a few things on a standalone test application, we manage to create a EGL context with GL 4.6 on an Nvidia + Linux + Proprietary drivers workstation. But we did not succeed on a AMD + Linux + Opensource drivers, only giving us a GL 3.0 context at most. So this is highly platform dependant and might need more work to get it working nicely. There is already an EGL backend to GHOST but it's targeted at GLES and outdated. So we can divide this into 2 tasks: - Create compatible GL > 3.3 context with EGL (with maximum platform compatibility). - Add offscreen context functions to EGL backends. This is not a really hard task but it is not a priority for now. So postponing for now. Here is the test application we used : [P671](https://archive.blender.org/developer/P671.txt) Thanks @Sergey for helping me on this one.

Vytenis commented

2019-01-07 15:27:41 +01:00

Added subscriber: @xvytis

Marek Olsak commented

2019-02-22 20:29:31 +01:00

Added subscriber: @mareko

Marek Olsak commented

2019-02-22 20:29:31 +01:00

AMD guy here.

You need to request the Core profile for older versions of our open source driver. If you don't, you'll get the compatibility profile which is mainly used by legacy applications.

The original plan was to keep the Core profile version at GL 4.5 and the Compatibility profile version at GL 3.0 forever, but we changed our minds and started adding Compatibility profile support last year:
Mesa 18.0: OpenGL 3.0
Mesa 18.1: OpenGL 3.1
Mesa 18.2: OpenGL 4.4
Mesa 18.3: OpenGL 4.5

You are shooting yourselves in the foot by using the Compatibility (default) profile. The Core profile is slightly faster if you are CPU-bound.

AMD guy here. You need to request the Core profile for older versions of our open source driver. If you don't, you'll get the compatibility profile which is mainly used by legacy applications. The original plan was to keep the Core profile version at GL 4.5 and the Compatibility profile version at GL 3.0 forever, but we changed our minds and started adding Compatibility profile support last year: Mesa 18.0: OpenGL 3.0 Mesa 18.1: OpenGL 3.1 Mesa 18.2: OpenGL 4.4 Mesa 18.3: OpenGL 4.5 You are shooting yourselves in the foot by using the Compatibility (default) profile. The Core profile is slightly faster if you are CPU-bound.

Will commented

2019-08-21 22:15:19 +02:00

Added subscriber: @WillW

noki paike commented

2019-09-22 16:45:24 +02:00

Added subscriber: @nokipaike

noki paike commented

2019-09-22 16:45:24 +02:00

as informative:
AMD Guy, Marek Olsak write on #69853 comments:

Mesa drivers now support the Compatibility profile on AMD hw starting with the AMD GCN architecture, so you don't need the Core profile anymore.

as informative: AMD Guy, Marek Olsak write on #69853 comments: > Mesa drivers now support the Compatibility profile on AMD hw starting with the AMD GCN architecture, so you don't need the Core profile anymore.

Evan Ryan commented

2019-10-30 22:11:08 +01:00

Added subscriber: @evantryan

getshi commented

2019-12-13 11:05:06 +01:00

Added subscriber: @getshi

getshi commented

2019-12-17 09:27:52 +01:00

I wanted to comment on the importance of this by giving an example.

I'm working in research, where we build prototypes of new simulation methods usually as stand-alone applications (because that's the fastest way, or because there's libraries to extend, etc.), and they are usually equipped with no or only trivial rendering. Writing them directly in tools such as blender is often too big of an overhead. However, previewing rendered animations of tons of simulations (done outside blender in prototypes with only simple or no rendering) can be vital; using blender's shader nodes, modifier stack, and general layouting would be an amazing tool for that. Eevee renderings can look amazing at a fraction of the cost of cycles renderings, and for the purpose of previewing any sort of animation this can be a crucial timesaver.

For now, I am stuck with rendering on my workstation, even though I would have multiple gpu machines available in a cluster but without displays.

I wanted to comment on the importance of this by giving an example. I'm working in research, where we build prototypes of new simulation methods usually as stand-alone applications (because that's the fastest way, or because there's libraries to extend, etc.), and they are usually equipped with no or only trivial rendering. Writing them directly in tools such as blender is often too big of an overhead. However, previewing rendered animations of *tons* of simulations (done outside blender in prototypes with only simple or no rendering) can be vital; using blender's shader nodes, modifier stack, and general layouting would be an amazing tool for that. Eevee renderings can look amazing at a fraction of the cost of cycles renderings, and for the purpose of previewing any sort of animation this can be a crucial timesaver. For now, I am stuck with rendering on my workstation, even though I would have multiple gpu machines available in a cluster but without displays.

Clément Foucault was unassigned by Dalai Felinto

2019-12-23 16:36:14 +01:00

Julian Eisel commented

2020-01-17 11:54:23 +01:00

Added subscriber: @JulianEisel

Julian Eisel commented

2020-01-17 11:54:23 +01:00

Bringing back EGL support and adding offscreen rendering support to it is being worked on here: D6585: X11 EGL context

Bringing back EGL support and adding offscreen rendering support to it is being worked on here: [D6585: X11 EGL context](https://archive.blender.org/developer/D6585)

Christian Rauch commented

2020-01-17 12:08:23 +01:00

Added subscriber: @christian.rauch

Christian Rauch commented

2020-01-17 12:08:23 +01:00

In #54638#851657, @JulianEisel wrote:
Bringing back EGL support and adding offscreen rendering support to it is being worked on here: D6585: X11 EGL context

Yes, that is definitely possible with EGL. I implemented that once for another project: https://github.com/stevenlovegrove/Pangolin/blob/master/src/display/device/display_headless.cpp

However, this should be done in a separate GHOST implementation which does not use any windowing system. This would then allow you to build Blender without any window system dependencies (X11 / Wayland) and run it on a server. I am not sure about GLX dependencies though, the GLEW package in Ubuntu is linked against GLX (libGLX.so).

> In #54638#851657, @JulianEisel wrote: > Bringing back EGL support and adding offscreen rendering support to it is being worked on here: [D6585: X11 EGL context](https://archive.blender.org/developer/D6585) Yes, that is definitely possible with EGL. I implemented that once for another project: https://github.com/stevenlovegrove/Pangolin/blob/master/src/display/device/display_headless.cpp However, this should be done in a separate GHOST implementation which does not use any windowing system. This would then allow you to build Blender without any window system dependencies (X11 / Wayland) and run it on a server. I am not sure about GLX dependencies though, the GLEW package in Ubuntu is linked against GLX (`libGLX.so`).

pampa nie commented

2020-02-26 00:30:23 +01:00

Added subscriber: @pampa

Shaw Vrana commented

2020-04-16 05:44:25 +02:00

Added subscriber: @svrana

Shaw Vrana commented

2020-04-16 05:44:28 +02:00

Removed subscriber: @svrana

Shaw Vrana commented

2020-04-16 05:44:35 +02:00

Added subscriber: @svrana

Vaclav Cermak commented

2020-06-15 17:17:44 +02:00

Added subscriber: @VaclavCermak

David Mikucki commented

2020-09-02 15:28:03 +02:00

Added subscriber: @davidmikucki

Raimund Klink commented

2020-11-20 00:15:00 +01:00

Added subscriber: @Raimund58

Mikhail Matrosov commented

2021-01-17 21:30:20 +01:00

Added subscriber: @ktdfly

Alan Melling commented

2021-03-29 18:18:33 +02:00

Added subscriber: @alan.melling

Daniel Houghton commented

2021-06-29 18:24:43 +02:00

Added subscriber: @DanielHoughton

Daniel Houghton commented

2021-06-29 18:24:43 +02:00

In #54638#829993, @getshi wrote:
I wanted to comment on the importance of this by giving an example.

I'm working in research, where we build prototypes of new simulation methods usually as stand-alone applications (because that's the fastest way, or because there's libraries to extend, etc.), and they are usually equipped with no or only trivial rendering. Writing them directly in tools such as blender is often too big of an overhead. However, previewing rendered animations of tons of simulations (done outside blender in prototypes with only simple or no rendering) can be vital; using blender's shader nodes, modifier stack, and general layouting would be an amazing tool for that. Eevee renderings can look amazing at a fraction of the cost of cycles renderings, and for the purpose of previewing any sort of animation this can be a crucial timesaver.

For now, I am stuck with rendering on my workstation, even though I would have multiple gpu machines available in a cluster but without displays.

I'll second that this would be an exciting feature to add to our headless render farm that is currently limited to Cycles only, and even there an artist who accidentally includes a grease pencil object in their render will crash the render attempt.

> In #54638#829993, @getshi wrote: > I wanted to comment on the importance of this by giving an example. > > I'm working in research, where we build prototypes of new simulation methods usually as stand-alone applications (because that's the fastest way, or because there's libraries to extend, etc.), and they are usually equipped with no or only trivial rendering. Writing them directly in tools such as blender is often too big of an overhead. However, previewing rendered animations of *tons* of simulations (done outside blender in prototypes with only simple or no rendering) can be vital; using blender's shader nodes, modifier stack, and general layouting would be an amazing tool for that. Eevee renderings can look amazing at a fraction of the cost of cycles renderings, and for the purpose of previewing any sort of animation this can be a crucial timesaver. > > For now, I am stuck with rendering on my workstation, even though I would have multiple gpu machines available in a cluster but without displays. I'll second that this would be an exciting feature to add to our headless render farm that is currently limited to Cycles only, and even there an artist who accidentally includes a grease pencil object in their render will crash the render attempt.

Manuel Rais commented

2021-07-27 15:08:10 +02:00

Added subscriber: @g-lul