Render API load from file does not load all layers #21410
Closed
opened 2010-03-01 22:54:18 +01:00 by Michal Mielczynski
·
47 comments
No Branch/Tag Specified
main
blender-v3.6-release
universal-scene-description
temp-sculpt-dyntopo
blender-v3.3-release
asset-browser-frontend-split
brush-assets-project
asset-shelf
anim/armature-drawing-refactor-3
temp-sculpt-dyntopo-hive-alloc
tmp-usd-python-mtl
tmp-usd-3.6
blender-v3.5-release
blender-projects-basics
blender-v2.93-release
temp-sculpt-attr-api
realtime-clock
sculpt-dev
gpencil-next
bevelv2
microfacet_hair
xr-dev
principled-v2
v3.6.3
v3.3.11
v3.6.2
v3.3.10
v3.6.1
v3.3.9
v3.6.0
v3.3.8
v3.3.7
v2.93.18
v3.5.1
v3.3.6
v2.93.17
v3.5.0
v2.93.16
v3.3.5
v3.3.4
v2.93.15
v2.93.14
v3.3.3
v2.93.13
v2.93.12
v3.4.1
v3.3.2
v3.4.0
v3.3.1
v2.93.11
v3.3.0
v3.2.2
v2.93.10
v3.2.1
v3.2.0
v2.83.20
v2.93.9
v3.1.2
v3.1.1
v3.1.0
v2.83.19
v2.93.8
v3.0.1
v2.93.7
v3.0.0
v2.93.6
v2.93.5
v2.83.18
v2.93.4
v2.93.3
v2.83.17
v2.93.2
v2.93.1
v2.83.16
v2.93.0
v2.83.15
v2.83.14
v2.83.13
v2.92.0
v2.83.12
v2.91.2
v2.83.10
v2.91.0
v2.83.9
v2.83.8
v2.83.7
v2.90.1
v2.83.6.1
v2.83.6
v2.90.0
v2.83.5
v2.83.4
v2.83.3
v2.83.2
v2.83.1
v2.83
v2.82a
v2.82
v2.81a
v2.81
v2.80
v2.80-rc3
v2.80-rc2
v2.80-rc1
v2.79b
v2.79a
v2.79
v2.79-rc2
v2.79-rc1
v2.78c
v2.78b
v2.78a
v2.78
v2.78-rc2
v2.78-rc1
v2.77a
v2.77
v2.77-rc2
v2.77-rc1
v2.76b
v2.76a
v2.76
v2.76-rc3
v2.76-rc2
v2.76-rc1
v2.75a
v2.75
v2.75-rc2
v2.75-rc1
v2.74
v2.74-rc4
v2.74-rc3
v2.74-rc2
v2.74-rc1
v2.73a
v2.73
v2.73-rc1
v2.72b
2.72b
v2.72a
v2.72
v2.72-rc1
v2.71
v2.71-rc2
v2.71-rc1
v2.70a
v2.70
v2.70-rc2
v2.70-rc
v2.69
v2.68a
v2.68
v2.67b
v2.67a
v2.67
v2.66a
v2.66
v2.65a
v2.65
v2.64a
v2.64
v2.63a
v2.63
v2.61
v2.60a
v2.60
v2.59
v2.58a
v2.58
v2.57b
v2.57a
v2.57
v2.56a
v2.56
v2.55
v2.54
v2.53
v2.52
v2.51
v2.50
v2.49b
v2.49a
v2.49
v2.48a
v2.48
v2.47
v2.46
v2.45
v2.44
v2.43
v2.42a
v2.42
v2.41
v2.40
v2.37a
v2.37
v2.36
v2.35a
v2.35
v2.34
v2.33a
v2.33
v2.32
v2.31a
v2.31
v2.30
v2.28c
v2.28a
v2.28
v2.27
v2.26
v2.25
Labels
Clear labels
This issue affects/is about backward or forward compatibility
Issues relating to security: https://wiki.blender.org/wiki/Process/Vulnerability_Reports
Apply labels
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
This issue affects/is about backward or forward compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest: Wayland
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Issues relating to security: https://wiki.blender.org/wiki/Process/Vulnerability_Reports
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest: Wayland
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
Milestone
Set milestone
Clear milestone
No items
No Milestone
Projects
Set Project
Clear projects
No project
Assignees
Assign users
Clear assignees
No Assignees
18 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#21410
Reference in New Issue
There is no content yet.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?
%%%When using NetworkRender mode, postprocessing with Depth of Field (DOF) node does not update focus lenght (ie. driven by animated object). Same scene rendered locally is working as expected.%%%
Changed status to: 'Open'
%%%May not be a problem with netrender per se, but Martin's probably best equipped to see what the issue is. Assigning to him.%%%
%%%Rendering goes fine, but compositing has to be turned off on the client, otherwise it reapplies composition that has been done on the slaves. I'm going to have to figure something out to turn if off automatically.%%%
%%%There's also this warning when loading back the result (a multilayer exr) in Blender through the render engine api, loading the exr manually works fine.
warning, channel with no rect set Composite.Combined.A
warning, channel with no rect set Composite.Combined.B
warning, channel with no rect set Composite.Combined.G
warning, channel with no rect set Composite.Combined.R
Assigning to brecht for testing (I think he's responsible for the render engine).%%%
%%%Adding a demo file.
Unzip, open blender from this directory (it will load results.exr from cwd), load result.blend, execute opened script (it should change the render engine to Test Render, do so if it didn't), render. It should display exr warnings in console, the layers mentioned in the warnings will be missing from the result buffer (but would be visible if loading the exr in the image editor manually).%%%
%%%Looked into this bug again, and the problem is that network render expects all layers in the .exr file to be loaded, while only the ones that the renderer expects are loaded. This is working as designed actually, but in this case it would be useful to support loading other layers too. Not sure yet how to solve without making major changes.%%%
%%%IMHO, when loading a multilayer exr as render result, it should recreate the result as it was originally (so that saving and reloading as result is a lossless operation).%%%
%%%Ehm... so the conclusion is, no bug but just 'how it was coded' ?%%%
%%%Depends what we want to support for render engines. Personally, I'd consider it a bug, at the very least a missing feature.%%%
%%%I guess it's not a bug in the render api but still one in network render, which is also a feature. Anyway, I'll try to fix this.%%%
%%%Reminder to this report, it's getting dusty! :)%%%
%%%I see that "[#28049] Blender "Network Render" does not render effects (nodes) from the Compositor." has been marked as a duplicate of this.
It would seem to me that this is an extremely serious bug as it is impossible to use netrender to farm out any scenes that use compositing nodes.
The slave nodes render the scene correctly with both the renderlayer and the composite output in the exr file. The master monitor web page correctly shows a thumbnail of the composited scene. However, the client seems to only expect, and will only display the renderlayer before compositing. It completely ignores the composite layer, as if it doesn't exist. The client does not appear to be expecting the composite layer at all, even though Compositing is checked in the Post Processing settings in the blend file.
This bug can be very easily verified, using the default cube. Simply open the nodes editor, enable nodes and set up a simple compositing pipeline (e.g. connect alpha from the renderlayer to the compositing node). Rendering locally will display the expected output, whilst a network render will only display the renderlayer before compositing.
This bug effectively renders (sorry!) the netrender absolutely useless apart from for very simple work. I am extremely puzzled as to why there appears to be no progress on this bug, since it is a year and a half old.
%%%
%%%Please note that the work around written in the other report still applies.
That is: the result exr files on the master can be fed into Blender's compositor as multilayer and that will work correctly (as if you'd render them locally).
But it sure would be nice to get it fixed.%%%
%%%Support of multilayer EXR files should've been improved in svn rev45358. probably it'll fix the tissue?
Brecht, othewise mayeb better to move this into a todo?%%%
%%%What is the current status? Can we close or mark as ToDo? %%%
%%%Closing this one due to no updates. %%%
Changed status from 'Open' to: 'Archived'
%%%Still have the problem%%%
%%%...problem presists. (Blender 2.58 r5365)
So, the current workflow to correct this is to render with the results appearing un-noded, and then wire them in again to the compositor as "Movie Clip" (image sequence) and set up again your nodes. Did all those Open Movies work this way?
It works I think. Bit of a headache to explain to a class of 30 or so students, but then I am assessing them on their ability to be able to discuss efficiencies in a pipeline so it's one more thing they could get marks for when they apply the workaround as long as they say what they did. I'm wondering if Loki has the same issues (although our school network hates Loki because it's Java - another story)%%%
%%%Still having issues with this in Blender 2.68.1 r58706.
For those who are frustrated with this bug, or use the compositor for allot of their work, here is the best work around that I have been able to find so far.
When you are ready to use Network Render, send it as a job, and let it render. Ounce the render is finished, you will need to find the EXR files that are sitting on the Master (the file path is located under the Master dialogue in Blender). At this point, open up a new instance of Blender, and bring up the composite node editor. delete the scene/render layer node, and add and image node and attach it to the composite node. Navigate to and add the first .EXR image in your rendered animation that is on the Master. Switch it from single image to image sequence and set the correct values for your animation. Make sure that the render settings for the scene match the blend file that you sent to the render farm originally, and then render out to the desired format.
It seems to me that this bug reduces the effectiveness of the idea of the render farm considerably. As composting usually equates to more render power needed, therefore more computing power is needed to get the final result faster(aka a render farm). Compositing seems like a priority to me, but my priorities don't always match someone elses.)
I appreciate all the hard work that has been put into this add-on, it has helped me with my work for over 3 years now. It's just been frustrating that I have been running into this problem, along with not being able to do full-sample motion-blur/anti-aliasing through Network Render (both processor intensive tasks), over the last 3 years.%%%
Added subscriber: @Calvin1602
Still present in 2.72b.
Please, at least re-open it ! This bug makes network rendering almost useless.
Network render addon is currently in orphaned state. For until new maintainer appears issues with it are considered a TODO hence should be handled outside of the bug tracker.
Added subscriber: @Dagamedia
This comment was removed by @Dagamedia
Hi; this is my first patch on Blender. Please bear with me, esp. if it's not the right place to do code reviews (IRC ?).
Despite what has been said elsewhere, all EXR layers are correctly loaded. They are just not linked to the RenderResult, because when it's created, it creates one layer (a RenderResult layer, not OpenEXR layer) per render layer, but doesn't create a layer for the composite. So when output.exr arrives from the network, Composite.Combined.A/R/G/B have nowhere to go and are simply put aside.
In my simple tests, this patches makes the following cases work :
Comments welcome !
patch21410.diff
Thanks for the patch, some of us have been waiting for a long time for something like this. I am planning on working with it soon.
I'm no expert, but it seems that this patch isn't format specific. Will it output PNG's correctly, according to the scenarios listed? This format is a bit more condusive for quick VFX workflows, as it supports transparency.
Thanks again!
Added subscriber: @JohanTriHandoyo
Hi Arnaud,
Thanks for your patch. Can you let me know how to apply this patch? Does this also works with .png output?
Hi johantri, I'm not sure what your question is. If you're uncomfortable using diff/patch you can just apply it by hand, editing render_result.c, and hope it didn't change too much in 1.5 years. Then re-compile Blender (which is too complicated to explain here).
Re-reading the code, it should work with any format (except maybe some exr variants ?). Please let me know if PNG doesn't work.
Added subscriber: @JamesCrowther
Still present in 2.78a , I guess this is not going away? We've encountered it whilst developing a network render addon similar to Netrender, except we do bucket rendering or frame splitting. For us, the ability read in a multilayer exr is rather critical since the work around by Arnoud above doesn;t work for us.
With each machine doing a section of a single frame, the ability to use the feature of a render result to stitch images back together is key, but we can only do this with single exr files or other flatter image formats like png, jpg.
Can one of the devs give us at least an indication as to whether it is planned to provide support for blender to allow a render result to read a multilayer open exr in such a way as to provide access to the passes in the compositor? It seems rather odd that this can be done using the image node and opening the same file that cannot be read in properly using a RenderResult.
Any suggestions for our particular situation would be most helpful too ;)
Hi James,
Not an actual answer, but what my patch does is (if I recall correctly, it's been 2 years) make it possible for each computer to composite its own render.
If you're doing frame splitting, this doesn't have any sense; you can't composite just a part of an image (think gaussian blur), you need the full frame. So you'll need to do the following on the master (or one of the clients):
No idea if Python has such low-level access on EXR files.
Hi Arnaud, :D
What we are missing here is the ability to import the merged exr as you say above. Blender allow us to merge results by using multiple render results and writing the pixels with the correct offsets. But we get the same error as Martin was getting (according to his posts above) when we try to use a multilayer exr file. This is the problem since without being able to import a multilayer exr, we can't access the passes of the render that are part of the exr file.
From what I can see, the problem arises due to the naming of the layers inside the exr file. I currently have no problem loading a single layer exr file using multiple render results and merging the split frames back together, easy!
The multilayer file has different names and hierarchy for the various 'channels'. In the single exr file (which works when loading using a render result, but only has the RBGA channels of the combines pass) there are only 4 channels called 'R', 'G', 'B' and 'A'. These are no problem for use with a render result, all works as documented.
However in the multilayer file, which has the separate passes (and which you can load into the compositor and see all your passes as mentioned above), there are the following 'channels'
Judging by the posts above, this isn't expected by the RenderResults' methods for loading layers. I think it expects just one layer per file and probably not with this naming convention either. Seems like an easy fix.
Anyway, we're working on a way around this in our project. Shouldn't take us too long and network rendering will be able to feed directly into the compositor just like the native engines do, if you're interested in learning more about what we're doing, just let me know and I'll pass you the details :D
Thanks again for reaching out, appreciate your time that you took in posting here for me!
a RenderResult contains a list of RenderLayer's (rr->layers)
a RenderLayer contains a list of RenderPass'es (rl->passes)
The naming is exactly what EXR wants. See
c359343f8d/source/blender/render/intern/source/render_result.c (L176)
Hi Arnaud, have been looking through this code, but not sure what you meant by your last comment. I am new to the blender cpp code. I did find this
c359343f8d/source/blender/render/intern/source/render_result.c (L1379)
which refers to loading data into a render result from an exr file. I am guessing the issue we're talking about here lies somewhere in this code rather than the code you mention since that code on L176 just seems to create the names used in the exr file.
I agree the naming is consistent with the exr format, just seems odd to me that blender writes the multilayer exr file, but then can not load that very same file back as a render result. I am guessing that this is due to the use case of external renderers not being fully supported right up into being able to use the multiple passes in the way the internal render engines do. Again all guesses.
In any case, thanks again for commenting. Gotta get back to it! So much coding to do!
Added subscriber: @brinux
Using 2.78c and just bought four new computers to use as my "render farm" and then found out from personal experience that network rendering ignores the compositor (tears) - and this has been broken for at least SEVEN YEARS?!?j
The compositor is critical - not having that effectively negates the render farm - and no, I'm not going to download thousands of individual EXR files and then run them through the compositor again, or look for faster ways of doing something the hard way. That's untenably time consuming, and obviously the opposite of what the render farm should be used for; that being: increased speed and efficiency.
What needs to happen is that the network renderer should return the images correctly, and that's it -- so I'm begging you guys, PLEASE fix this. SEVEN YEARS, PLEASE.
To those working through this: the fastest way to use my render farm now is to abandon network rendering and load the .blend file on each machine manually, then point the outputs on all machines to the same network share location while turning off the "overwrite" box in the Output - still a bother, but massively faster than the EXR workaround.
I'm sooo looking forward to this long-awaited fix. 2.78d ?
@brinux We're working on an addon to fix this, anticipate that the fix will be integrated into our addon end of June this year. Feel free to contact us if you'd like to test. We got snagged on this problem too but found there is a way round it, but it involves programming, so we're building this into our addon which does network rendering.
Thanks!
James
crowdrender
Added subscriber: @fablefox
Since this was previously closed because no updates/activity, opening it again since now there are some activity/interest.
how do we go around this, TODO? @brecht
Changed status from 'Archived' to: 'Open'
@JamesCrowther Thank you for your response, and test invite. Yes please! I would be happy to test.
I'm an advanced blender user, though I may still need a bit of coaching in how the testing process works (assuming there's procedures, reporting steps etc.).
Feel free to contact me here or at brian@angryeggplant.com
Changed status from 'Open' to: 'Archived'
This is still a To Do item regardless of the amount of user interest. The render API was never designed for this purpose, and the network render addon is currently unmaintained.
Anyone is free to submit a patch to fix this. The above attached patch is not correct though, it should not always create extra composite render layers when rendering with nodes enabled, doing it at that point can break various things. Only the render API layer loading function should do it.
https://developer.blender.org/differential/diff/create/
Added subscriber: @Rabit
Hello Guys, just found quite the same during working on BlendNet addon (https://github.com/state-of-the-art/BlendNet/issues/57#issuecomment-687360342), it's not critical - but definitely will be great to have it fixed.