GPU: Improve GLSL workflow #127983
Labels
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset System
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Code Documentation
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Viewport & EEVEE
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Asset Browser Project
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Module
Viewport & EEVEE
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Severity
High
Severity
Low
Severity
Normal
Severity
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#127983
Loading…
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Traversing the GLSL codebase is difficult for developers outside the Viewport & EEVEE module.
Even for well versed developers it can be annoying and time consuming.
For instance, to find a variable definition one needs to search for all occurrences in a directory and find the relevant occurrence of a variable name.
This is why I was thinking of allowing intellisense to compile the GLSL files as C++ header files for linting and symbol recognitions (go to definition, rename symbol, autocomplete).
I started by just changing some glsl file extension to
.glsl.hh
and see how it goes. Below is a list of things that does not work and how we could change it.Note that not all of these are needed to make it usable. Even with some errors, many intellisense features works (i.e. go to definition) and would already be useful.
in/inout/out keywords:
These are easy to bypass with a simple define. The C++ will compile fine even with everything passed by copy.
GLSL builtin functions:
Things like
dot
,greaterThan
,gl_VertexID
should be declared appropriately as stubs. This is a bit much work but can be done incrementally.Array definition:
The GLSL definition syntax for arrays is not standard. It is not widespread in the codebase so either we live with it generating linting errors or we rewrite these array in a way that fits both language.
Opaque types:
Textures and images should have studs as well as the sampling functions. Should be easy to emulate.
Include directives:
All dependencies are currently expressed using
#pragma BLENDER_REQUIRE
. This is not compatible with C++. I had some vague idea of doing a real include system for GLSL sources in the past but it might be out of scope.Simply renaming all the pragmas to
#include
needs a few things:#pragma once
on every lib file#include
as it did for#pragma BLENDER_REQUIRE
.#pragma once
and#include
directives (they conflict with a GLSL extension).Vector swizzles:
First I wanted to reuse the
blender::math::VecBase
class but it lacks support for swizzling as member syntax. The usage of swizzling is quite widespread and letting it fail would be quite annoying. So I propose to have a very thin vector/matrix API that only do the type checking, no actual implementation needed.#127982 is an experiment of what would it take to support this.
External resources:
This is the most annoying one to get working. This is where I think I regret having chosen the CreateInfo object oriented definition interface paradigm.
External resources are maybe the most obfuscated thing in our GLSL development.
One way would be to declare stubs in global space. But this would just duplicates the create infos inside the shader with no way to sync the two.
Edit: After some testing, fclem/blender#3 seems to be a good compromise.
Tasks
#include
Changes #128076in/out: Is similar how cycles does it. So proven, easy to add. but.... can result to issues as some platforms are more forgiven. (don't know any better solution for now, this is already the case)
arrays: can be make this backend specific. By default using macros that lead to lintable code.
I would prioritize to get all the stuff to get a simple shader working (GPU_BUILTINS?) and add features to increase the shaders that will work. It seems like most tasks are already ordered in this way. I can imagine that the complex shaders can only be compiled when everything mentioned above is implemented.
I do think "Blender-GLSL" will be the source of the shaders for still a long time. (Personal idea): In the future GLSL might be compiled to SpirV (also for MSL) as more platforms are moving into this direction. More tools will exist for authoring SpirV as well. Has not much to do with this task.
Still unsure how and if we can benefit from what slang is doing. There is a lot of investments in this shader language and if promised are kept could be a in-place replacement for GLSL.
Turns out I have the stubs part mostly solved in Malt (they're auto-generated from the spec), so we could reuse that:
https://github.com/bnpr/Malt/blob/Development/scripts/build_intellisense_glsl.py
https://github.com/bnpr/Malt/blob/Development/Malt/Shaders/Intellisense/intellisense.glsl
It doesn't cover math operators and vector swizzles (my main goal was function autocompletion) but you have that part already figured out.
Replacing the
CreateInfo
workflow seems like a huge task, I'm not sure how feasible it is or whether we should get rid of it just to get autocompletion fully working.If the main goal is to help non-viewport devs, providing stubs for the code that they're more likely to work with (I guess this is mostly about the overlay shaders?) seems like a more reasonable approach.
For file extensions, most editors allow adding extra file associations, so I would consider taking that approach instead.
I see a lot of duplication and it might not lint some type conversions errors. But that's a valuable reference. Thanks!
That's a good point.
That's correct. That's why it is at the bottom of the list. It needs more design.
About the Array definition syntax, I propose defining the following:
Reasoning about using
floatX
,uintX
and the like instead ofvecX
anduvec2
is that we want eventually to move all the codebase to the same typename as our C++ codebase. So this would be our first move into this direction.But if we prefer not having this discrepancy for now and rename eveything later, we coulduse the GLSL type names.
About the external resources, I found a not so intrusive system. Here is the mockup:
If multiple interfaces are required, they can be defined in their own file and be inherited.
This should allow functionality to be kepts as-is and only require some mass renaming.
Bonus: The
USE_INTERFACE
macro could be use to check at shader compile time that the dependencies are fulfilled. It would automatically inject stuff like this:This would remove the defines from the C++ CreateInfo declaration, right?
How would libraries that require resources work?
If it's up to the user code to include their requirements then the intellisense doesn't work in the library file, but otherwise the C++ side won't work.
Maybe libraries should be included from the info file itself?
This is more subjective, but having one file for each CreateInfo seems quite annoying.
Maybe in some cases we could do something like:
I am not sure I understand this. If you mean it will replace the need for this kind of declaration:
[[readonly]] [[storage_buffer]] [[binding(0)]] ViewCullingData view_culling_buf[DRW_VIEW_LEN];
Then yes. All resources and uniform declaration would stay inside the
_info.hh
files.Inside the libraries you would need only one line
USE_INTERFACE(your_create_info_name)
. It would propagate to users of that library.I don't understand this statement. Why would intellisense be broken if we include the create info?
We keep the same file structure as we have right now. The first lib to include the create info includes it (
#pragma once
). This include would be skipped by GLSL or be noop.This was my first idea but that doesn't look good.
Dependencies between create infos can be done using
using
inside of the namespace. I tested and this works on all compilers. However, this requires some code reorganization as currently dependencies are not defined in order. But I think we would benefit from a clearer dependency structure.I mean the
ShaderCreate::define
function, AFAIK we can't have a#define DEFINE(macro) #define macro
.But you would also need an
ADDITIONAL_INFO(library_info_name)
from the user CreateInfo.So, do we have to declare everything twice and it's up to us to ensure it's in sync?
At the moment we usually have multiple
CreateInfo
declarations per _info file. 🤔Namespaces in GLSL?
I think it will be better to discuss around a working prototype as there is clearly some misunderstanding 😄.
The namespace are only for the C++ linting.
All the GLSL pipeline for shader compilation would work exactly the same as they do right now (generated interface at runtime).
You have a point on this. I don't think it is possible either. Eventually, it would be best to avoid most of these defines and rely on
specialization_constant
instead. So, maybe that is not that important to support.Also it is only really problematic for defines that have values (since the others are just checked by
#ifdef
and co.). There are 153 out of 390 macro definition that fall into this category.There is no "user create info". The create info will only be defined once, exactly where they are currently.
Sounds familiar. 😅
@pragma37
d69613426c
is an example of usage. It now works as expected. I need to finish some things for it to work fully but the simple case is working.See the PR for more detail fclem/blender#3 (check individual commits).
It is quite a lot of macro. But I think the benefit outweigh the obfuscation.