Cryptomatte layers saved incorrectly with EXR DWA compression #96933

Closed
opened 2022-03-31 23:46:27 +02:00 by Daniel Salazar · 25 comments
Member

System Information
Operating system: Windows-10-10.0.19044-SP0 64 Bits
Graphics card: NVIDIA GeForce GTX 1080 with Max-Q Design/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 512.15

Blender Version
Broken: version: 3.2.0 Alpha, branch: master, commit date: 2022-03-30 23:25, hash: 8466fd4bd1

Description of error
Cryptomatte layers on a multilayer EXR (DWA) file will be saved as lossy which makes them unusable. Lossy compression in DWA is only meant for RGB layers.

Lossless cryptomatte (ZIP)
image.png
Lossy Cryptomatte (DWAB)
image.png

Apparently DWA handles these type of exceptions based on hardcoded layer names .

There's also the possibility of setting custom rules in the file. More info can be read in the commented section at EXR's ImfDwaCompressor.cpp file.

The rules for deciding which strategy to use are setup in initializeDefaultChannelRules().
When writing a file, the relevant rules needed to decode are written into the start of the data block, making a self-contained file.
If initializeDefaultChannelRules() doesn't quite suite your naming conventions, you can adjust the rules without breaking decoder compatability.

Cryptomatte example file in DWAB
Cryptomatte Example.zip

**System Information** Operating system: Windows-10-10.0.19044-SP0 64 Bits Graphics card: NVIDIA GeForce GTX 1080 with Max-Q Design/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 512.15 **Blender Version** Broken: version: 3.2.0 Alpha, branch: master, commit date: 2022-03-30 23:25, hash: `8466fd4bd1` **Description of error** Cryptomatte layers on a multilayer EXR (DWA) file will be saved as lossy which makes them unusable. Lossy compression in DWA is only meant for RGB layers. Lossless cryptomatte (ZIP) ![image.png](https://archive.blender.org/developer/F12960547/image.png) Lossy Cryptomatte (DWAB) ![image.png](https://archive.blender.org/developer/F12960549/image.png) Apparently DWA handles these type of exceptions based on [hardcoded layer names ](https://github.com/Psyop/Cryptomatte/issues/35#issuecomment-331042950). There's also the possibility of setting custom rules in the file. More info can be read in the commented section at EXR's [ImfDwaCompressor.cpp ](https://github.com/AcademySoftwareFoundation/openexr/blob/2205245599709d88013b7da03b176c1c73c8aedb/src/lib/OpenEXR/ImfDwaCompressor.cpp) file. > The rules for deciding which strategy to use are setup in initializeDefaultChannelRules(). > When writing a file, the relevant rules needed to decode are written into the start of the data block, making a self-contained file. > If initializeDefaultChannelRules() doesn't quite suite your naming conventions, you can adjust the rules without breaking decoder compatability. Cryptomatte example file in DWAB [Cryptomatte Example.zip](https://archive.blender.org/developer/F12961208/Cryptomatte_Example.zip)
Author
Member

Added subscriber: @zanqdo

Added subscriber: @zanqdo

Added subscriber: @emilstefx

Added subscriber: @emilstefx

Added subscriber: @TomsSeglins

Added subscriber: @TomsSeglins

Added subscriber: @iss

Added subscriber: @iss

Changed status from 'Needs Triage' to: 'Needs User Info'

Changed status from 'Needs Triage' to: 'Needs User Info'

Can you add simple .blend file?

Can you add simple .blend file?
Author
Member

In #96933#1333369, @iss wrote:
Can you add simple .blend file?

A simple blend file of what?

> In #96933#1333369, @iss wrote: > Can you add simple .blend file? A simple blend file of what?
Member

Added subscriber: @PratikPB2123

Added subscriber: @PratikPB2123

Changed status from 'Needs User Info' to: 'Confirmed'

Changed status from 'Needs User Info' to: 'Confirmed'

Thanks for file. I wasn't sure how to check, since output did look correct.

Thanks for file. I wasn't sure how to check, since output did look correct.
Contributor

Added subscriber: @Raimund58

Added subscriber: @Raimund58

Added subscriber: @SteffenD

Added subscriber: @SteffenD
Sergey Sharybin self-assigned this 2022-08-31 16:03:11 +02:00

This issue was referenced by cb9b6cefb3

This issue was referenced by cb9b6cefb3c01d3f759a16252aca59fa4607dd24

This issue was referenced by df751516e1

This issue was referenced by df751516e1f64bbc78b02582c631c3cdd0bc032d

Changed status from 'Confirmed' to: 'Resolved'

Changed status from 'Confirmed' to: 'Resolved'

This issue was referenced by 40194f7219

This issue was referenced by 40194f72191fc629b96ce619540b3f0f5da0ccfc

This issue was referenced by 06005b0870

This issue was referenced by 06005b0870be9a0a3b73b4c388c26988f1f991d2

Blender Version - 4.0.0 Beta

Just wanted to mention that this issue still appears to exist when writing an EXR with DWA compression from the File Output node.

If you save a multichannel EXR from the render buffer using image > save as , the file is written correctly and the Cryptomatte channel is not compressed. I've included a simple example to illustrate this.

However, If the same render is saved via the FileOutput node, the cryptomatte channel is compressed like the initial example provided by @zanqdo.

I see there's some discussion about channel naming allowing this to flag if a channel is deemed compression worthy? Perhaps there is some trick to making this work with specific naming from the FileOutput node? As the channel naming structure is technically different between the images.

google group discussion
psyop discussion

I am using Nuke to read the resulting EXR.

Blender Version - 4.0.0 Beta Just wanted to mention that this issue still appears to exist when writing an EXR with DWA compression from the File Output node. If you save a multichannel EXR from the render buffer using `image > save as` , the file is written correctly and the Cryptomatte channel is not compressed. I've included a simple example to illustrate this. However, If the same render is saved via the FileOutput node, the cryptomatte channel is compressed like the initial example provided by @zanqdo. I see there's some discussion about channel naming allowing this to flag if a channel is deemed compression worthy? Perhaps there is some trick to making this work with specific naming from the FileOutput node? As the channel naming structure is technically different between the images. [google group discussion](https://groups.google.com/g/alshaders/c/JJVoG-c_5O0/m/piaYgpmbAQAJ) [psyop discussion](https://github.com/Psyop/Cryptomatte/issues/35) I am using Nuke to read the resulting EXR.

Unfortunately, there is no trick to achieve lossless output for those passes in the file output node without modifying the code.

Ideally the node will somehow detect passes which needs to be stored lossless. Although, it might also be good to implement a checkbox on the node inputs to allow lossless storage of generated data.

Unfortunately, there is no trick to achieve lossless output for those passes in the file output node without modifying the code. Ideally the node will somehow detect passes which needs to be stored lossless. Although, it might also be good to implement a checkbox on the node inputs to allow lossless storage of generated data.
Author
Member

Unfortunately, there is no trick to achieve lossless output for those passes in the file output node without modifying the code.

Ideally the node will somehow detect passes which needs to be stored lossless. Although, it might also be good to implement a checkbox on the node inputs to allow lossless storage of generated data.

That would be quite welcomed

> Unfortunately, there is no trick to achieve lossless output for those passes in the file output node without modifying the code. > > Ideally the node will somehow detect passes which needs to be stored lossless. Although, it might also be good to implement a checkbox on the node inputs to allow lossless storage of generated data. That would be quite welcomed

Hey there
DWAA and DWAB codecs are still bugged indeed, however you can you Pxr24 (lossy) codec as well - it seems to work with cryptomate information correctly.

Hey there DWAA and DWAB codecs are still bugged indeed, however you can you Pxr24 (lossy) codec as well - it seems to work with cryptomate information correctly.
Contributor

@Peter-Kmet If you found a new bug, please open a new report with clear reproducing steps.

@Peter-Kmet If you found a new bug, please open a new report with clear reproducing steps.

@Peter-Kmet If you found a new bug, please open a new report with clear reproducing steps.

Hey @Raimund58 - its not a new bug - just existing one wasn't fixed yet (Cryptomatte layers saved incorrectly with EXR DWA compression - its clearly descriped at the begining of this thread)
I found some "workaround" to get small file size and usable cryptomatte pass that may help someone else as well.

> @Peter-Kmet If you found a new bug, please open a new report with clear reproducing steps. Hey @Raimund58 - its not a new bug - just existing one wasn't fixed yet (Cryptomatte layers saved incorrectly with EXR DWA compression - its clearly descriped at the begining of this thread) I found some "workaround" to get small file size and usable cryptomatte pass that may help someone else as well.
Contributor

@Peter-Kmet The bug was fixed long ago except for File Output nodes and the "save as" dialog. So, if you are still experiencing it for normal renders, please open a new bug report.

@Peter-Kmet The bug was fixed long ago except for File Output nodes and the "save as" dialog. So, if you are still experiencing it for normal renders, please open a new bug report.

@Peter-Kmet The bug was fixed long ago except for File Output nodes and the "save as" dialog. So, if you are still experiencing it for normal renders, please open a new bug report.

Thanks for letting me know - I experience it in File Output node setup.

> @Peter-Kmet The bug was fixed long ago except for File Output nodes and the "save as" dialog. So, if you are still experiencing it for normal renders, please open a new bug report. Thanks for letting me know - I experience it in File Output node setup.
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
11 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#96933
No description provided.