image datablocks fail to import multilayer exr layers that have a dot in the name. #89991

New Issue

michael campbell · 2021-07-20T15:05:05+02:00

michael campbell commented

2021-07-20 15:05:05 +02:00

System Information
Operating system: Windows-10-10.0.19041-SP0 64 Bits
Graphics card: GeForce GTX 1070/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 461.92

Blender Version
Broken: version: 3.0.0 Alpha, branch: master, commit date: 2021-07-16 19:29, hash: 118803893e
Worked: (newest version of Blender that worked as expected)

Exact steps for others to reproduce the error

open this file in the Blender image editor or an image node in the compositor:

The multilayer EXR contains the following layers as shown in Affinity:

when opening the file in Blender, all EXR layers that contain a dot in the name fail, as shown below. The image node is missing one of the EXR layers:

**System Information** Operating system: Windows-10-10.0.19041-SP0 64 Bits Graphics card: GeForce GTX 1070/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 461.92 **Blender Version** Broken: version: 3.0.0 Alpha, branch: master, commit date: 2021-07-16 19:29, hash: `118803893e` Worked: (newest version of Blender that worked as expected) **Exact steps for others to reproduce the error** open this file in the Blender image editor or an image node in the compositor: ![dots.dont.work.exr](https://archive.blender.org/developer/F10229921/dots.dont.work.exr) The multilayer EXR contains the following layers as shown in Affinity: ![image.png](https://archive.blender.org/developer/F10229941/image.png) when opening the file in Blender, all EXR layers that contain a dot in the name fail, as shown below. The image node is missing one of the EXR layers: ![image.png](https://archive.blender.org/developer/F10229944/image.png)

michael campbell commented

2021-07-20 15:05:05 +02:00

Added subscriber: @3di

Philipp Oeser commented

2021-07-20 15:22:52 +02:00

Added subscriber: @lichtwerk

Philipp Oeser commented

2021-07-20 15:22:52 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Philipp Oeser commented

2021-07-20 15:22:52 +02:00

Can confirm.

Philipp Oeser commented

2021-07-20 15:30:51 +02:00

#71574 (OpenEXR layers from View Layers with names containing a period can't be read by Nuke) is related (if not a duplicate)

michael campbell commented

2021-07-20 15:40:13 +02:00

Probably not a duplicate, I guess that indicates Nuke also has the same bug....maybe they're both using the same function of the openexr library to import, whereas Affinity isn't?

Philipp Oeser commented

2021-07-20 15:57:05 +02:00

Hm, here is a similar file that actually seems to work:

Rendered from:
#71574.blend

[philippo@lic3 build_linux_lite]$ exrheader /lichtwerk/01_projects/0072_BORG_BlenderFoundationTracker/BORG_Produktion/08_Bugtracker_reports/T89991/dots.do.work.exr 

file /lichtwerk/01_projects/0072_BORG_BlenderFoundationTracker/BORG_Produktion/08_Bugtracker_reports/T89991/dots.do.work.exr:

file format version: 2, flags 0x400
BlenderMultiChannel (type string): "Blender V2.55.1 and newer"
Camera (type string): "Camera"
Date (type string): "2021/07/20 15:46:33"
File (type string): "/lichtwerk/01_projects/0072_BORG_BlenderFoundationTracker/BORG_Produktion/08_Bugtracker_reports/T89991/T71574.blend"
Frame (type string): "1"
RenderTime (type string): "00:00.18"
Scene (type string): "Scene"
Time (type string): "00:00:00:01"
channels (type chlist):
    Composite.Combined.A, 32-bit floating-point, sampling 1 1
    Composite.Combined.B, 32-bit floating-point, sampling 1 1
    Composite.Combined.G, 32-bit floating-point, sampling 1 1
    Composite.Combined.R, 32-bit floating-point, sampling 1 1
    name no dots.Combined.A, 32-bit floating-point, sampling 1 1
    name no dots.Combined.B, 32-bit floating-point, sampling 1 1
    name no dots.Combined.G, 32-bit floating-point, sampling 1 1
    name no dots.Combined.R, 32-bit floating-point, sampling 1 1
    name no dots.Depth.Z, 32-bit floating-point, sampling 1 1
    name.with.dots.Combined.A, 32-bit floating-point, sampling 1 1
    name.with.dots.Combined.B, 32-bit floating-point, sampling 1 1
    name.with.dots.Combined.G, 32-bit floating-point, sampling 1 1
    name.with.dots.Combined.R, 32-bit floating-point, sampling 1 1
    name.with.dots.Depth.Z, 32-bit floating-point, sampling 1 1
compression (type compression): zip, multi-scanline blocks
cycles.name no dots.render_time (type string): "00:00.08"
cycles.name no dots.samples (type string): "4"
cycles.name no dots.synchronization_time (type string): "00:00.00"
cycles.name no dots.total_time (type string): "00:00.09"
cycles.name.with.dots.render_time (type string): "00:00.08"
cycles.name.with.dots.samples (type string): "4"
cycles.name.with.dots.synchronization_time (type string): "00:00.00"
cycles.name.with.dots.total_time (type string): "00:00.08"
dataWindow (type box2i): (0 0) - (959 539)
displayWindow (type box2i): (0 0) - (959 539)
lineOrder (type lineOrder): increasing y
pixelAspectRatio (type float): 1
screenWindowCenter (type v2f): (0 0)
screenWindowWidth (type float): 1
type (type string): "scanlineimage"

[philippo@lic3 build_linux_lite]$ exrheader /lichtwerk/01_projects/0072_BORG_BlenderFoundationTracker/BORG_Produktion/08_Bugtracker_reports/T89991/dots.dont.work.exr 

file /lichtwerk/01_projects/0072_BORG_BlenderFoundationTracker/BORG_Produktion/08_Bugtracker_reports/T89991/dots.dont.work.exr:

file format version: 2, flags 0x0
BlenderMultiChannel (type string): "Blender V2.55.1 and newer"
Date (type string): "2021/07/20 14:00:03"
File (type string): "E:\programming\blender\my addons\incomplete\auto_file_output\blend files\test_upstream.blend"
Scene (type string): "Scene"
channels (type chlist):
    name no dots.A, 32-bit floating-point, sampling 1 1
    name no dots.B, 32-bit floating-point, sampling 1 1
    name no dots.G, 32-bit floating-point, sampling 1 1
    name no dots.R, 32-bit floating-point, sampling 1 1
    name.with.dots.A, 32-bit floating-point, sampling 1 1
    name.with.dots.B, 32-bit floating-point, sampling 1 1
    name.with.dots.G, 32-bit floating-point, sampling 1 1
    name.with.dots.R, 32-bit floating-point, sampling 1 1
compression (type compression): none
dataWindow (type box2i): (0 0) - (1919 1079)
displayWindow (type box2i): (0 0) - (1919 1079)
lineOrder (type lineOrder): increasing y
pixelAspectRatio (type float): 1
screenWindowCenter (type v2f): (0 0)
screenWindowWidth (type float): 1
type (type string): "scanlineimage"

Hm, here is a similar file that actually seems to work: ![dots.do.work.exr](https://archive.blender.org/developer/F10229971/dots.do.work.exr) Rendered from: [#71574.blend](https://archive.blender.org/developer/F10229972/T71574.blend) ``` [philippo@lic3 build_linux_lite]$ exrheader /lichtwerk/01_projects/0072_BORG_BlenderFoundationTracker/BORG_Produktion/08_Bugtracker_reports/T89991/dots.do.work.exr file /lichtwerk/01_projects/0072_BORG_BlenderFoundationTracker/BORG_Produktion/08_Bugtracker_reports/T89991/dots.do.work.exr: file format version: 2, flags 0x400 BlenderMultiChannel (type string): "Blender V2.55.1 and newer" Camera (type string): "Camera" Date (type string): "2021/07/20 15:46:33" File (type string): "/lichtwerk/01_projects/0072_BORG_BlenderFoundationTracker/BORG_Produktion/08_Bugtracker_reports/T89991/T71574.blend" Frame (type string): "1" RenderTime (type string): "00:00.18" Scene (type string): "Scene" Time (type string): "00:00:00:01" channels (type chlist): Composite.Combined.A, 32-bit floating-point, sampling 1 1 Composite.Combined.B, 32-bit floating-point, sampling 1 1 Composite.Combined.G, 32-bit floating-point, sampling 1 1 Composite.Combined.R, 32-bit floating-point, sampling 1 1 name no dots.Combined.A, 32-bit floating-point, sampling 1 1 name no dots.Combined.B, 32-bit floating-point, sampling 1 1 name no dots.Combined.G, 32-bit floating-point, sampling 1 1 name no dots.Combined.R, 32-bit floating-point, sampling 1 1 name no dots.Depth.Z, 32-bit floating-point, sampling 1 1 name.with.dots.Combined.A, 32-bit floating-point, sampling 1 1 name.with.dots.Combined.B, 32-bit floating-point, sampling 1 1 name.with.dots.Combined.G, 32-bit floating-point, sampling 1 1 name.with.dots.Combined.R, 32-bit floating-point, sampling 1 1 name.with.dots.Depth.Z, 32-bit floating-point, sampling 1 1 compression (type compression): zip, multi-scanline blocks cycles.name no dots.render_time (type string): "00:00.08" cycles.name no dots.samples (type string): "4" cycles.name no dots.synchronization_time (type string): "00:00.00" cycles.name no dots.total_time (type string): "00:00.09" cycles.name.with.dots.render_time (type string): "00:00.08" cycles.name.with.dots.samples (type string): "4" cycles.name.with.dots.synchronization_time (type string): "00:00.00" cycles.name.with.dots.total_time (type string): "00:00.08" dataWindow (type box2i): (0 0) - (959 539) displayWindow (type box2i): (0 0) - (959 539) lineOrder (type lineOrder): increasing y pixelAspectRatio (type float): 1 screenWindowCenter (type v2f): (0 0) screenWindowWidth (type float): 1 type (type string): "scanlineimage" [philippo@lic3 build_linux_lite]$ exrheader /lichtwerk/01_projects/0072_BORG_BlenderFoundationTracker/BORG_Produktion/08_Bugtracker_reports/T89991/dots.dont.work.exr file /lichtwerk/01_projects/0072_BORG_BlenderFoundationTracker/BORG_Produktion/08_Bugtracker_reports/T89991/dots.dont.work.exr: file format version: 2, flags 0x0 BlenderMultiChannel (type string): "Blender V2.55.1 and newer" Date (type string): "2021/07/20 14:00:03" File (type string): "E:\programming\blender\my addons\incomplete\auto_file_output\blend files\test_upstream.blend" Scene (type string): "Scene" channels (type chlist): name no dots.A, 32-bit floating-point, sampling 1 1 name no dots.B, 32-bit floating-point, sampling 1 1 name no dots.G, 32-bit floating-point, sampling 1 1 name no dots.R, 32-bit floating-point, sampling 1 1 name.with.dots.A, 32-bit floating-point, sampling 1 1 name.with.dots.B, 32-bit floating-point, sampling 1 1 name.with.dots.G, 32-bit floating-point, sampling 1 1 name.with.dots.R, 32-bit floating-point, sampling 1 1 compression (type compression): none dataWindow (type box2i): (0 0) - (1919 1079) displayWindow (type box2i): (0 0) - (1919 1079) lineOrder (type lineOrder): increasing y pixelAspectRatio (type float): 1 screenWindowCenter (type v2f): (0 0) screenWindowWidth (type float): 1 type (type string): "scanlineimage" ```

michael campbell commented

2021-07-20 16:11:52 +02:00

hmm, in your file it doesn't show an output per layer, instead it changes the outputs based on the chosen layer. The functionality that my addon relies on, is to show all layers as outputs, otherwise it'll break a couple of months worth of programming.

hmm, in your file it doesn't show an output per layer, instead it changes the outputs based on the chosen layer. The functionality that my addon relies on, is to show all layers as outputs, otherwise it'll break a couple of months worth of programming. ![image.png](https://archive.blender.org/developer/F10229992/image.png)

michael campbell commented

2021-07-20 16:12:25 +02:00

i'm outputting the file from the compositor file output node by the way, whereas you're outputting it from a render animation.

Philipp Oeser commented

2021-07-20 16:34:17 +02:00

Added subscribers: @Jeroen-Bakker, @brecht

Philipp Oeser commented

2021-07-20 16:34:17 +02:00

Problem seems to be the assumption in imb_exr_split_channel_name / imb_exr_split_token that the whole name contains three tokens:

/* last token is channel identifier */
...
/* second token is pass name */
...
/* all preceding tokens combined as layer name */

see https://developer.blender.org/diffusion/B/browse/master/source/blender/imbuf/intern/openexr/openexr_api.cpp$1418
In your case, there is only a channel and pass, but no layer detected.
If that is the case, imb_exr_begin_read_mem will create a layer via imb_exr_get_layer (should this be called imb_exr_layer_ensure then?) -- and somewhere this seems to go wrong

Unsure how this should work:

should pass and layer be the same?
should we enforce a pass (e.g. "Combined") in this case?
should a layer and channels be permitted (without a pass)?

CC @Jeroen-Bakker
CC @brecht

Problem seems to be the assumption in `imb_exr_split_channel_name` / `imb_exr_split_token` that the whole name contains three tokens: ``` /* last token is channel identifier */ ... /* second token is pass name */ ... /* all preceding tokens combined as layer name */ ``` see https://developer.blender.org/diffusion/B/browse/master/source/blender/imbuf/intern/openexr/openexr_api.cpp$1418 In your case, there is only a `channel` and `pass`, but no `layer` detected. If that is the case, `imb_exr_begin_read_mem` will **create** a layer via `imb_exr_get_layer` (should this be called `imb_exr_layer_ensure` then?) -- and somewhere this seems to go wrong Unsure how this **should** work: - should `pass` and `layer` be the same? - should we enforce a `pass` (e.g. "Combined") in this case? - should a `layer` and `channels` be permitted (without a `pass`)? CC @Jeroen-Bakker CC @brecht

michael campbell commented

2021-07-20 16:35:03 +02:00

very strange:

Here's with all inputs not having dots. No option to choose a layer in the image node, they just all appear as outputs (which is imperative):

But if I change one of the inputs to have dots, then I get an option to choose layers, but only a single layer exists, and the letter after the final dot is taken to be the output...

I think maybe someone has gotten confused when adding multilayer to the file output node, and have taken inputs and layers to be the same thing. The correct functionality is to be able to specify the same layer for multiple inputs....although I'm not sure that explains the dot problem

very strange: Here's with all inputs not having dots. No option to choose a layer in the image node, they just all appear as outputs (which is imperative): ![image.png](https://archive.blender.org/developer/F10230010/image.png) But if I change one of the inputs to have dots, then I get an option to choose layers, but only a single layer exists, and the letter after the final dot is taken to be the output... ![image.png](https://archive.blender.org/developer/F10230016/image.png) I think maybe someone has gotten confused when adding multilayer to the file output node, and have taken inputs and layers to be the same thing. The correct functionality is to be able to specify the same layer for multiple inputs....although I'm not sure that explains the dot problem

michael campbell commented

2021-07-20 16:40:48 +02:00

This comment was removed by @3di

*This comment was removed by @3di*

michael campbell commented

2021-07-20 16:42:09 +02:00

This comment was removed by @3di

*This comment was removed by @3di*

michael campbell commented

2021-07-20 17:04:07 +02:00

OK, I found the problem. It's the file output node not adding any layer to the name:

it should be:

Layer . Name. File_format

so it's working when rendered from ctrl f12, because the output is:

name.with.dots.Combined.RGB

whereas the file output node is outputting:

name.with.dots.RGB (missing the layer)

so if the file output node isn't going to let the user specify a layer to assign the inputs to, then it should just add a layer name that doesn't contain any dots to the start of the string.

OK, I found the problem. It's the file output node not adding any layer to the name: it should be: Layer . Name. File_format so it's working when rendered from ctrl f12, because the output is: name.with.dots.Combined.RGB whereas the file output node is outputting: name.with.dots.RGB (missing the layer) so if the file output node isn't going to let the user specify a layer to assign the inputs to, then it should just add a layer name that doesn't contain any dots to the start of the string.

Philipp Oeser commented

2021-07-20 17:15:37 +02:00

Changed status from 'Confirmed' to: 'Needs Developer To Reproduce'

Philipp Oeser commented

2021-07-20 17:15:37 +02:00

so if the file output node isn't going to let the user specify a layer to assign the inputs to, then it should just add 'Combined' between the name and the file_format.

either that, or one of these (permitting layer.channel - this is according to spec - and enforcing layer.pass.channel when reading -- or permit layer.channel as well which might be a bigger change since assumption atm is layer.pass.channel ):

In #89991#1193782, @lichtwerk wrote:
Unsure how this should work:

should pass and layer be the same?

should we enforce a pass (e.g. "Combined") in this case?

should a layer and channels be permitted (without a pass)?

CC @Jeroen-Bakker
CC @brecht

At this point, I think it would be good to get input from module devs.

> so if the file output node isn't going to let the user specify a layer to assign the inputs to, then it should just add 'Combined' between the name and the file_format. either that, or one of these (permitting `layer.channel` - this is according to spec - and enforcing `layer.pass.channel` when **reading** -- or permit `layer.channel` as well which might be a bigger change since assumption atm is `layer.pass.channel` ): > In #89991#1193782, @lichtwerk wrote: > Unsure how this **should** work: > - should `pass` and `layer` be the same? > - should we enforce a `pass` (e.g. "Combined") in this case? > - should a `layer` and `channels` be permitted (without a `pass`)? > > CC @Jeroen-Bakker > CC @brecht At this point, I think it would be good to get input from module devs.

michael campbell commented

2021-07-20 17:27:03 +02:00

yes, it should definitely follow the same convention as the rest of blender:

layer.pass.channel

as oppose to the current :

pass.channel

yes, it should definitely follow the same convention as the rest of blender: layer.pass.channel as oppose to the current : pass.channel

michael campbell commented

2021-07-20 17:28:54 +02:00

I can just manually add something without dots to the beginning of the input name for now. Would be great to have the file output node do that automatically though.

michael campbell commented

2021-07-20 17:45:43 +02:00

nope, still breaks if the pass has dots in it because the function that creates the image blocks when opening EXR's is searching from the right of the string, splitting it to get the channel, then searching from the right again and splitting it to get the pass, and then one more time to get the layer.

In order for it to work if the pass contains dots, the code that opens multilayer exr's it needs to do this instead:

x = full_layer_name
channel = x[ rfind('.') : ]
layer = x[ : x.find('.') ]
pass = x[ x.find('.') : x.rfind('.') ]

nope, still breaks if the pass has dots in it because the function that creates the image blocks when opening EXR's is searching from the right of the string, splitting it to get the channel, then searching from the right again and splitting it to get the pass, and then one more time to get the layer. In order for it to work if the pass contains dots, the code that opens multilayer exr's it needs to do this instead: x = full_layer_name channel = x[ rfind('.') : ] layer = x[ : x.find('.') ] pass = x[ x.find('.') : x.rfind('.') ]

Brecht Van Lommel commented

2021-07-20 18:12:59 +02:00

First, I would recommend to not use dots in the names at all, as they are the standard separator for EXR and it will inevitably lead to problems in Blender or other software. On top of that, while the EXR specification mentions dots, a lot of other software deviates from that and uses underscores instead. So I would use neither.

For reading, the best Blender can do is tweak heuristics to get a better result in undefined cases. For writing, we could show a warning and/or replace the dots when writing EXR files to sidestep the problem.

When only two tokens are provided when reading an EXR file, they are pass and channel. These should not be interpreted as layer.

First, I would recommend to not use dots in the names at all, as they are the standard separator for EXR and it will inevitably lead to problems in Blender or other software. On top of that, while the EXR specification mentions dots, a lot of other software deviates from that and uses underscores instead. So I would use neither. For reading, the best Blender can do is tweak heuristics to get a better result in undefined cases. For writing, we could show a warning and/or replace the dots when writing EXR files to sidestep the problem. When only two tokens are provided when reading an EXR file, they are `pass` and `channel`. These should not be interpreted as `layer`.

michael campbell commented

2021-07-20 18:22:18 +02:00

Would it be possible to keep everything as it is, but just use the code I posted above, then it won't matter if there are dots in the pass name, and it'll continue to work under all circumstances it currently works for. In the case of there being only two tokens, then you could just have:

if pass == channel:

  pass = layer
  layer = None

I'm basically saving out AOV's via a file output node. If the user duplicates an AOV and doesn't realise Blender's auto renaming convention of adding .### after a duplicate AOV name, then Blender won't be able to re-open the saved EXRs.

Alternatively, the auto duplicate renaming could use _### instead of .###

Would it be possible to keep everything as it is, but just use the code I posted above, then it won't matter if there are dots in the pass name, and it'll continue to work under all circumstances it currently works for. In the case of there being only two tokens, then you could just have: if pass == channel: ``` pass = layer layer = None ``` I'm basically saving out AOV's via a file output node. If the user duplicates an AOV and doesn't realise Blender's auto renaming convention of adding .### after a duplicate AOV name, then Blender won't be able to re-open the saved EXRs. Alternatively, the auto duplicate renaming could use _### instead of .###

michael campbell commented

2021-07-20 18:24:26 +02:00

other software doesn't matter, the files generated will only ever be used by Blender.

michael campbell commented

2021-07-20 18:25:58 +02:00

although the above alternative method I posted of getting the tokens from the string would also enable a user to import from other softwares that have dots in the pass name.

michael campbell commented

2021-07-20 18:49:52 +02:00

it would break again if the view layer has dots in it. Maybe the safest option is to enforce no dots in the aov and view layer names. Can't believe exr stores variables in the string, I thought it would use something like exr or json.

Brecht Van Lommel commented

2021-07-20 19:44:24 +02:00

The current heuristic works well for Blender in many cases because the standard channel and pass names do not contain dots. The view layer name on the other hand is more likely to have dots, and is parsed correctly if you go from right to left. So I think that's pretty reasonable.

Avoiding automatic dots and warning against manually added dots in AOV names in Blender seems like a practical thing we can do.

The current heuristic works well for Blender in many cases because the standard channel and pass names do not contain dots. The view layer name on the other hand is more likely to have dots, and is parsed correctly if you go from right to left. So I think that's pretty reasonable. Avoiding automatic dots and warning against manually added dots in AOV names in Blender seems like a practical thing we can do.

blender-admin commented

2021-10-19 12:13:27 +02:00

This issue was referenced by 8e8932c8ff

This issue was referenced by 8e8932c8ff28ec1b1ac7af9254244d94b326b9b8

Brecht Van Lommel commented

2021-10-19 13:09:43 +02:00

Changed status from 'Needs Developer To Reproduce' to: 'Resolved'

Brecht Van Lommel closed this issue

2021-10-19 13:09:43 +02:00

Brecht Van Lommel self-assigned this 2021-10-19 13:09:43 +02:00

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

image datablocks fail to import multilayer exr layers that have a dot in the name. #89991