GPU: Swap Buffers Design #111389

Closed
Jeroen Bakker wants to merge 1 commits from Jeroen-Bakker:gpu/gpu-swap-buffers into main

When changing the target branch, be careful to rebase the branch in your fork to match. See documentation.
Member

In the current implementation window manager directly calls GHOST
swap buffers. The GPU backend state could become invalid as it is
not aware of the buffer swap.

NOTE: This should move to documentation, as the implementation
has done in a different PR.

Solution 1

Introduce GPU_context_swap_buffers so the backend can
do pre/post stuff to ensure the state is always valid.

sequenceDiagram
	box
      participant GPU_context_swap_buffers
      participant GPUContext
      participant VKContext
    end
    box
      participant GHOST_Context
    end
    
	activate GPU_context_swap_buffers
    GPU_context_swap_buffers->>GPUContext: swap_buffers
    activate GPUContext
    
    GPUContext->>VKContext: swap_buffers_begin
    activate VKContext
    deactivate VKContext
    
    GPUContext->>GHOST_Context: swap_buffers
    activate GHOST_Context
    deactivate GHOST_Context
    
    GPUContext->>VKContext: swap_buffers_end
    activate VKContext
    deactivate VKContext
    
    deactivate GPUContext
    deactivate GPU_context_swap_buffers

Reasoning

In Vulkan the texture layout is not query-able, but the application is
responsible to always communicate the correct layout to the Vulkan API.
This isn't working for images from the swap chain.

In Vulkan the command buffer needs to be in a certain state as GHOST and
the GPU module share the same instance. Separating the command buffers
could be done, but would have synchronization issues as there can still
be command in flight or not, which isn't known by GHOST.

During start of blender contexts are initialized twice that leads to
incorrect behaviors as the state is not clear.
During start of blender buffers are swapped without submitting the
command buffer, leading to missing texture clears.

Vulkan

VKContext::swap_buffers_begin can then make sure that the texture
of the default framebuffer is set to what GHOST expects (GENERAL)
it can also flush all commands that are still in recording state.
this includes the barrier to change the framebuffer texture layout.

VKContext::swap_buffers_end can than retrieve the new framebuffer,
command buffer, texture from GHOST and reset what is needed to reset.

Solution 2

Use backend specific callbacks as to keep the issue located in the
vulkan backend. Similar to MTL, except we might still need to have
pre/post callbacks.

sequenceDiagram
    box 
      participant GHOST_SwapWindowBuffers
      participant GHOST_ContextVK
    end
    box
      participant VKContext
    end
    box
      participant Vulkan
    end
    
	activate GHOST_SwapWindowBuffers
    GHOST_SwapWindowBuffers->>GHOST_ContextVK: swapBuffers
    activate GHOST_ContextVK
    
    GHOST_ContextVK->>VKContext: swap_buffers_pre_callback
    activate VKContext
    Note over VKContext: Ensure command buffer flushed and framebuffer texture format.
    deactivate VKContext

    GHOST_ContextVK->>Vulkan: multiple commands
    activate Vulkan
    Note over GHOST_ContextVK, Vulkan: Perform present and related commands.
    deactivate Vulkan
    
    GHOST_ContextVK->>VKContext: swap_buffers_post_callback
    activate VKContext
    Note over VKContext: Retrieve updated swapchain handlers and recreate framebuffer, restart command buffer
    deactivate VKContext
   
    
    deactivate GHOST_ContextVK
    deactivate GHOST_SwapWindowBuffers

Reasoning

Control flow of the swap chain currently isn't clear in Vulkan,
Solution 1 might close the gab a bit, but is a solution that
would only add value to vulkan. So what is the value of adding
it to the public API.

GHOST Context and GPU are highly connected, but would still use
callbacks to keep them a bit separated.

Using a single callback would require giving control away from
GHOST_swapBuffers to the GPU module, which leads to potential
invalid state in GHOST. Currently not an issue, but best to avoid.

Main difference with Metal is that the Metal backend only draws
into the swapchain texture, during the present callback. Until
then it uses an intermediate. The Vulkan backend does draw
directly onto a swapchain texture, and swap buffers performs
a swap.

Solution 3

Similar to Metal where it is only allowed to draw/blit to the swap
chain texture in the scope of the present callback.

sequenceDiagram
    box 
      participant GHOST_SwapWindowBuffers
      participant GHOST_ContextVK
    end
    box
      participant VKContext
    end
    box
      participant Vulkan
    end
    
	activate GHOST_SwapWindowBuffers
    GHOST_SwapWindowBuffers->>GHOST_ContextVK: swapBuffers
    activate GHOST_ContextVK
    
    GHOST_ContextVK->>VKContext: swap_buffers_callback
    activate VKContext
    Note over VKContext: Flush command buffer, Blit framebuffer to swap chain texture
    deactivate VKContext

    GHOST_ContextVK->>Vulkan: multiple commands
    activate Vulkan
    Note over GHOST_ContextVK, Vulkan: Perform present and related commands.
    deactivate Vulkan
    
    deactivate GHOST_ContextVK
    deactivate GHOST_SwapWindowBuffers

Reasoning

Is less complex as both the GPU module and GHOST have their own
responsibility. Requires additional texture and blitting overhead.
In Metal swapchain are part of commands, in vulkan swap chain are
done on queue level.

NOTE: this solution will have allocated 3 times the required
texture memory to present a screen to the user. 1 are the per
area offscreen buffers/viewports, 1 the framebuffer and 1 the
swapchain texture.

It would improve the performance as the display texture can
be acquired later.

Conclusion

We will align the vulkan implementation with the metal one.
Having a single callback would reduce the complexity and
needed resources owned by the GHOST context.

In the current implementation window manager directly calls GHOST swap buffers. The GPU backend state could become invalid as it is not aware of the buffer swap. NOTE: This should move to documentation, as the implementation has done in a different PR. # Solution 1 Introduce `GPU_context_swap_buffers` so the backend can do pre/post stuff to ensure the state is always valid. ```mermaid sequenceDiagram box participant GPU_context_swap_buffers participant GPUContext participant VKContext end box participant GHOST_Context end activate GPU_context_swap_buffers GPU_context_swap_buffers->>GPUContext: swap_buffers activate GPUContext GPUContext->>VKContext: swap_buffers_begin activate VKContext deactivate VKContext GPUContext->>GHOST_Context: swap_buffers activate GHOST_Context deactivate GHOST_Context GPUContext->>VKContext: swap_buffers_end activate VKContext deactivate VKContext deactivate GPUContext deactivate GPU_context_swap_buffers ``` **Reasoning** In Vulkan the texture layout is not query-able, but the application is responsible to always communicate the correct layout to the Vulkan API. This isn't working for images from the swap chain. In Vulkan the command buffer needs to be in a certain state as GHOST and the GPU module share the same instance. Separating the command buffers could be done, but would have synchronization issues as there can still be command in flight or not, which isn't known by GHOST. During start of blender contexts are initialized twice that leads to incorrect behaviors as the state is not clear. During start of blender buffers are swapped without submitting the command buffer, leading to missing texture clears. **Vulkan** `VKContext::swap_buffers_begin` can then make sure that the texture of the default framebuffer is set to what GHOST expects (GENERAL) it can also flush all commands that are still in recording state. this includes the barrier to change the framebuffer texture layout. `VKContext::swap_buffers_end` can than retrieve the new framebuffer, command buffer, texture from GHOST and reset what is needed to reset. # Solution 2 Use backend specific callbacks as to keep the issue located in the vulkan backend. Similar to MTL, except we might still need to have pre/post callbacks. ```mermaid sequenceDiagram box participant GHOST_SwapWindowBuffers participant GHOST_ContextVK end box participant VKContext end box participant Vulkan end activate GHOST_SwapWindowBuffers GHOST_SwapWindowBuffers->>GHOST_ContextVK: swapBuffers activate GHOST_ContextVK GHOST_ContextVK->>VKContext: swap_buffers_pre_callback activate VKContext Note over VKContext: Ensure command buffer flushed and framebuffer texture format. deactivate VKContext GHOST_ContextVK->>Vulkan: multiple commands activate Vulkan Note over GHOST_ContextVK, Vulkan: Perform present and related commands. deactivate Vulkan GHOST_ContextVK->>VKContext: swap_buffers_post_callback activate VKContext Note over VKContext: Retrieve updated swapchain handlers and recreate framebuffer, restart command buffer deactivate VKContext deactivate GHOST_ContextVK deactivate GHOST_SwapWindowBuffers ``` **Reasoning** Control flow of the swap chain currently isn't clear in Vulkan, Solution 1 might close the gab a bit, but is a solution that would only add value to vulkan. So what is the value of adding it to the public API. GHOST Context and GPU are highly connected, but would still use callbacks to keep them a bit separated. Using a single callback would require giving control away from GHOST_swapBuffers to the GPU module, which leads to potential invalid state in GHOST. Currently not an issue, but best to avoid. Main difference with Metal is that the Metal backend only draws into the swapchain texture, during the present callback. Until then it uses an intermediate. The Vulkan backend does draw directly onto a swapchain texture, and swap buffers performs a swap. # Solution 3 Similar to Metal where it is only allowed to draw/blit to the swap chain texture in the scope of the present callback. ```mermaid sequenceDiagram box participant GHOST_SwapWindowBuffers participant GHOST_ContextVK end box participant VKContext end box participant Vulkan end activate GHOST_SwapWindowBuffers GHOST_SwapWindowBuffers->>GHOST_ContextVK: swapBuffers activate GHOST_ContextVK GHOST_ContextVK->>VKContext: swap_buffers_callback activate VKContext Note over VKContext: Flush command buffer, Blit framebuffer to swap chain texture deactivate VKContext GHOST_ContextVK->>Vulkan: multiple commands activate Vulkan Note over GHOST_ContextVK, Vulkan: Perform present and related commands. deactivate Vulkan deactivate GHOST_ContextVK deactivate GHOST_SwapWindowBuffers ``` **Reasoning** Is less complex as both the GPU module and GHOST have their own responsibility. Requires additional texture and blitting overhead. In Metal swapchain are part of commands, in vulkan swap chain are done on queue level. NOTE: this solution will have allocated 3 times the required texture memory to present a screen to the user. 1 are the per area offscreen buffers/viewports, 1 the framebuffer and 1 the swapchain texture. It would improve the performance as the display texture can be acquired later. # Conclusion We will align the vulkan implementation with the metal one. Having a single callback would reduce the complexity and needed resources owned by the GHOST context.
Jeroen Bakker added this to the 4.0 milestone 2023-08-22 16:05:34 +02:00
Jeroen Bakker added the
Interest
Vulkan
label 2023-08-22 16:05:34 +02:00
Jeroen Bakker self-assigned this 2023-08-22 16:05:34 +02:00
Jeroen Bakker added 1 commit 2023-08-22 16:05:46 +02:00
6db8db4940 GPU: Swap Buffers
In the current implementation window manager directly calls GHOST
swap buffers. The GPU backend state could become invalid as it is
not aware of the buffer swap.

This PR will introduce GPU_context_swap_buffers so the backend can
do pre/post stuff to ensure the state is always valid.

* Reasoning *

In Vulkan the texture layout is not query-able, but the application is
responsible to always communicate the correct layout to the Vulkan API.
This isn't working for images from the swap chain.

In Vulkan the command buffer needs to be in a certain state as GHOST and
the GPU module share the same instance. Separating the command buffers
could be done, but would have synchronization issues as there can still
be command in flight or not, which isn't known by GHOST.

During start of blender contexts are initialized twice that leads to
incorrect behaviors as the state is not clear.
During start of blender buffers are swapped without submitting the
command buffer, leading to missing texture clears.

* Things to keep in mind *

- MTLBackend introduced begin/end frame. I think it is still required as
  swapbuffers are optional when using offscreen contexts
Jeroen Bakker changed title from GPU: Swap Buffers to WIP: GPU: Swap Buffers 2023-08-22 16:06:20 +02:00
Jeroen Bakker added this to the EEVEE & Viewport project 2023-08-22 16:06:24 +02:00
First-time contributor

This looks like it would certainly be useful!
One feature that exists within the Metal backend/GHOST_ContextCGL currently is the present callback, which allows the GPU backend to specify a function to call in order to perform a present.

i.e:

metalRegisterPresentCallback

For Metal, this is required to achieve the same thing. Rather than the GHOST_* module being responsible for presenting, instead the GPU backend is.

in Metal: GHOST_ContextCGL::metalSwapBuffers() calls directly into void present(MTLRenderPassDescriptor *blit_descriptor, id<MTLRenderPipelineState> blit_pso, id<MTLTexture> swapchain_texture, id<CAMetalDrawable> drawable) which exists within the GPU module, in order to achieve the same end result as a swap_buffers_begin/end implementation.

In this case however, there is only a single function allowing the backend full control to coordinate everything, with the drawable being handed back from the GHOST module.

I'm not sure if it would then be easier to generalise the implementation to just a single callback where the GPU backend is responsible for the full presentation?

I guess this perhaps makes sense for Metal as there is only one possible GHOST context per API, whereas I imagine this is not true for Vulkan, where there are several different window manager implementations.

Anyway, I think the design looks good, but just throwing some thoughts out there.

For Metal, the existing mechanism could instead just be moved into either of the begin/end functions, and the actual present could be a no-op. This is also often a requirement in Metal, as the actual present can be called from within a dependent command buffer e.g. [command_buffer present], hence for synchronisation (to resolve old flickering issues), this had to be done this way before.

This looks like it would certainly be useful! One feature that exists within the Metal backend/GHOST_ContextCGL currently is the present callback, which allows the GPU backend to specify a function to call in order to perform a present. i.e: `metalRegisterPresentCallback` For Metal, this is required to achieve the same thing. Rather than the GHOST_* module being responsible for presenting, instead the GPU backend is. in Metal: `GHOST_ContextCGL::metalSwapBuffers()` calls directly into `void present(MTLRenderPassDescriptor *blit_descriptor, id<MTLRenderPipelineState> blit_pso, id<MTLTexture> swapchain_texture, id<CAMetalDrawable> drawable)` which exists within the GPU module, in order to achieve the same end result as a swap_buffers_begin/end implementation. In this case however, there is only a single function allowing the backend full control to coordinate everything, with the drawable being handed back from the GHOST module. I'm not sure if it would then be easier to generalise the implementation to just a single callback where the GPU backend is responsible for the full presentation? I guess this perhaps makes sense for Metal as there is only one possible GHOST context per API, whereas I imagine this is not true for Vulkan, where there are several different window manager implementations. Anyway, I think the design looks good, but just throwing some thoughts out there. For Metal, the existing mechanism could instead just be moved into either of the begin/end functions, and the actual present could be a no-op. This is also often a requirement in Metal, as the actual present can be called from within a dependent command buffer e.g. `[command_buffer present]`, hence for synchronisation (to resolve old flickering issues), this had to be done this way before.
Author
Member

Thanks for the feedback. I think that a callback might become to platform specific. vulkan would require different arguments, datatypes etc. But will look in more depth to the Mtl implementation before deciding what is best

Thanks for the feedback. I think that a callback might become to platform specific. vulkan would require different arguments, datatypes etc. But will look in more depth to the Mtl implementation before deciding what is best
Contributor

First of all, the swap-chain acquirer function is not constrained by any image-layout, nor does it perform transitions.
The layout simply needs to be VK_IMAGE_LAYOUT_PRESENT_SRC_KHR when presenting.

Regarding synchronization.

When I first implemented it in my branch, I synced using semaphores.

source

Here's how it works:

The swap-buffer function is divided into two parts: acquirer and present.

Then call the acquirer function on the GHOST side and VKContext gets the image-available-semaphores. And set the previous Semaphore to wait-semaphore of the first submission of the main loop.

Then, I did the final submission with signal-semaphore from GPU-Module, and wait-Semaphore of present's submission.

By doing this, we can be assured that we have acquired the image of that time. This is because the Semaphore and swap-chain-image are linked.

As for the commands, we don't need to synchronize them at all. So, I think the commands for present should be dedicated.

But then the official branch was significantly rewritten, so I made it a different implementation. At this stage, synchronization can be done without using semaphore. Also, I was able to implement resize.

However, in this case, we wait image-available-semaphores at present, so it is difficult to say that it is being used very effectively.
source

Either way, I can't find a clear reason why we have to share commands. Conversely, using the same command as the offscreen render for present submission increases the probability of leading to a serious error.

To clarify synchronization, the system provided by Vulkan is semaphore.

First of all, the swap-chain acquirer function is not constrained by any image-layout, nor does it perform transitions. The layout simply needs to be VK_IMAGE_LAYOUT_PRESENT_SRC_KHR when presenting. Regarding synchronization. When I first implemented it in my branch, I synced using semaphores. [source](https://projects.blender.org/vnapdv/tmp-vulkan/src/branch/first_time_startup/intern/ghost/intern/GHOST_ContextVK.cpp) Here's how it works: > The swap-buffer function is divided into two parts: acquirer and present. >> Then call the acquirer function on the GHOST side and `VKContext` gets the image-available-semaphores. And set the previous Semaphore to wait-semaphore of the first submission of the main loop. >>> Then, I did the final submission with signal-semaphore from GPU-Module, and wait-Semaphore of present's submission. `By doing this, we can be assured that we have acquired the image of that time. This is because the Semaphore and swap-chain-image are linked.` As for the commands, we don't need to synchronize them at all. So, I think the commands for present should be dedicated. But then the official branch was significantly rewritten, so I made it a different implementation. At this stage, synchronization can be done without using semaphore. Also, I was able to implement resize. However, in this case, we wait image-available-semaphores at present, so it is difficult to say that it is being used very effectively. [source](https://projects.blender.org/vnapdv/blender/src/branch/vk_query/intern/ghost/intern/GHOST_ContextVK.cc) Either way, I can't find a clear reason why we have to share commands. Conversely, using the same command as the offscreen render for present submission increases the probability of leading to a serious error. To clarify synchronization, the system provided by Vulkan is semaphore.
Contributor

Regarding tracking Image-Layout
Ideally, swap-buffer should be systematic with GHOST*. I think the problem now is the instability of that Image-Layout. However, I think that the actual blender drawing system is wonderful, so even if the layout is hidden in the dark during the development stage, I think that a robust layout transition structure will be visible.
So, I think that it is an effective means to track the layout for the time being.

PS. I think that Layout's GENERAL means Layout when using Storage-Image rather than general purpose.
In the case of Swap-Chain-Image, it is often the target of Attachment and Blit, but I don't think it will be Storage-Image.

Regarding tracking Image-Layout Ideally, swap-buffer should be systematic with GHOST*. I think the problem now is the instability of that Image-Layout. However, I think that the actual blender drawing system is wonderful, so even if the layout is hidden in the dark during the development stage, I think that a robust layout transition structure will be visible. So, I think that it is an effective means to track the layout for the time being. PS. I think that Layout's GENERAL means Layout when using Storage-Image rather than general purpose. In the case of Swap-Chain-Image, it is often the target of Attachment and Blit, but I don't think it will be Storage-Image.
Jeroen Bakker closed this pull request 2023-08-29 14:37:42 +02:00
Jeroen Bakker changed title from WIP: GPU: Swap Buffers to GPU: Swap Buffers Design 2023-08-29 14:37:57 +02:00

Pull request closed

Sign in to join this conversation.
No reviewers
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#111389
No description provided.