was required (compositor can also be used with only image input) told
the pipeline not to render when no composite nodes were available at all,
and with button 'Use Nodes' on.
- added a zoom factor for Planar Environment Map, allowing to zoom in or
out on the reflected object. The zoom is corrected when rendering, so it
just gives control over the resolution.
- Planar maps don't cycle anymore, but only render the front plane, using
standard image texture extend mode.
- threaded rendering is back in envmap render
now the button works as meant to be.
Still working on the feature though... I noticed exr files dont read
back when only part was saved (on user break). Working...
New option "Save Buffers", in first Output panel of renderbuttons, will not
allocate all render buffers, but instead save the rendered tiles to exr.
For each scene rendered, a single exr file then is created.
After rendering, the files get read, and only then the memory allocation is
done.
The exr files are saved in the temp dir (from user settings), and have
names derived from the filename+scene name. That way these buffers remain
relatively unique, and can be re-used later too.
Saving all render-layers and passes in a single file (as F3 command) will
be done later. Also reading back the current muli-layer exr files is not
supported yet (will read black). The purpose is that these files then can
be used as input for the Compositor.
One fun thing I added; after rendering once with this option, close
Blender, and restart it. If you have a Composite set up press 'R' on an
active RenderResult node. This will refresh the node(s) and load the exr,
so you can composite again without a re-render.
- Bug fix: the upper tile in a collumn for Panorama render didn't put the
mainthread to sleep properly. Now panorama renders 25% faster if you had
set Y-Parts to 4.
- Enabling Compositing in Scene for first time now adds a "Composite" node
too, so render output gets applied.
- An attempt to render with "Do Composite" without "Composite" node will
throw an error and stops rendering. In background mode it will just not
render at all, and print errors.
- Errors that prevent rendering now give a popup menu again.
- Having MBlur or Fields option on will now normally render, but with an
error print in console (not done yet...)
also could use correction for it.
The current perspective projected blur would look in 180 degree view like
this:
http://www.blender.org/bf/p2.jpg
(circle of planes rotating around camera)
After some fight with my rusty highschool gonio I got it fixed; nice
cylindrical projected speedvectors:
http://www.blender.org/bf/p1.jpg
The old implementation was added quite hackish (talking about 10 yr ago).
You also had to make a small image slice, which was extended Xparts in
size. That also required to adjust the camera angle. Very clumsy.
Now; when enabling the Panorama option, it will automatically apply the
panorama effect on the vertically aligned tiles. You can just enable or
disable the "Pano" button, to get a subtle lens effect like this:
(without pano)
http://www.blender.org/bf/rt.jpg
(with pano)
http://www.blender.org/bf/rt1.jpg
For Panorama render, the minimum slice size has been hardcoded to be 8
pixels. The XParts button goes up to 512 to allow that. In practice,
rendering 64 slices will already give very good images for a wide angle
lens of 90 degrees, the curvature of straight lines then is equal to
a circle of 256 points.
Rendering a full 360 degree panorama you do by creating an extreme wide
angle camera. The theory says camera-lens 5 should do 360 degrees, but
for some reason my tests reveil it's 5.1... there's a rounding error
somewhere, maybe related to the clipping plane start? Will look at that
later. :)
Also note that for each Xpart slice, the entire database needs to be
rotated around camera to correct for panorama, on huge scenes that might
give some overhead.
Threaded render goes fine for Panorama too, but it can only render the
vertically aligned parts in parallel. For the next panorama slice it has
to wait for all threads of the current slice to be ready.
On reading old files, I convert the settings to match as closely as
possible the new situation.
Since I cannot bump up the version #, the code detects for old panorama
by checking for the image size. If image width is smaller than height, it
assumes it's an old file (only if Panoroma option was set).
For some reason I thought SDL thread handling would be much simpler... but
the migration to posix pthread went very smooth and painless. Less code
even, and I even notice a slight performance increase!
All threading code is still wrapped in blenlib/intern/threads.c
Only real change was making the callback functions to return void pointer,
instead of an int.
The mutex handling is also different... there's no test anymore if a
mutex was initialized, which is a bit confusing. But it appears to run
all fine still. :)
Nathan Letwory has been signalled already to provide the Windows pthread
library and make/scons linking. For MSVC we might need help from someone
else later though.
the ones that get changed within threads, to communicate with the main
thread.
(Part of the long quest to get threaded render safe, especially in Linux)
passes in single file. Code is currently disabled, commit is mainly to
have a nicer method of excluding OpenEXR dependency from render module.
This should compile with disabled WITH_OPENEXR too.
Reason why EXR is great to include by default in Blender is its feature
to store unlimited layers and channels, and write this tile based. I
need the feature for saving memory; while rendering tiles, all full-size
buffers for all layers and passes are kept in memory now, which can go
into 100s of MB easily.
The code I commit now doesn't allocate these buffers while rendering, but
saves the tiles to disk. In the end is it read back. Overhead for large
renders (like 300 meg buffers) is 10-15 seconds, not bad.
Two more interesting aspects:
- Blender can save such multi-layer files in the temp directory, storing
it with .blend file name and scene name. That way, on each restart of Blender,
or on switching scenes, these buffers can be read. So you always see what was
rendered last. Also great for compositing work.
- This can also become an output image type for rendering. There's plenty of
cases where you want specific layers or passes saved to disk for later use.
Anyhoo, finishing it is another days of work, and I got more urgent stuff
now!
This now is a post-process option only (used to be in render).
It is only handled within the Imbuf/ module, on conversions from float
to byte rect, which atm mostly happens on saving images.
- Small fix: when using Scene RenderLayer nodes, the speed vectors for
these nodes were not created when that scene had "Do Composite" off.
(NOTE: new include dependency in Render module, might need MSVC update!
It has to include the imbuf/intern/openexr/ directory in search path)
-> New Composite node: "Hue Saturation".
Works like the former 'post process' menu. There's no gamma, brightness or
multiply needed in this node, for that the Curves Node functions better.
-> Enabled Toolbox in Node editor
This now also replaces the SHIFT+A for adding nodes. The nodes are
automatically added to the menus, using the 'class' category from the
type definition.
Current classes are (compositor examples):
Inputs: RenderResult, Image
Outputs: Composite, Viewer
Color Ops: RGB Curves, Mix, Hue Saturation, AlphaOver
Vector Ops: Normal, Vector Curves, Map Value
Filters: Filter, Blur, VectorBlur
Convertors: ColorRamp, RGBtoBW, Separate RGBA, Separate HSVA, Set Alpha
Generators: RGB, Value, Time
Groups: the list of custom defined nodes
-> OpenEXR tile saving support
Created an API for for saving tile-based Images with an unlimited amount
of layers/channels. I've tested it for 'render result' now, with the idea
that this can (optionally) replace the current inserting of tiles in the
main result buffers. Especially with a lot of layers, the used memory for
these buffers can easily go into the 100s of megs.
Two other advantages:
- all 'render result' layers can be saved entirely in a single file, for
later use in compositing, also for animation output.
- on each render, per scene, a unique temp file can be stored, allowing
to re-use these temp files on starting Blender or loading files, showing
the last result of a render command.
The option is currently disabled, needs more work... but I had to commit
this because of the rest of the work I did!
-> Bug fix
The Image node didn't call an execute event when browsing another image.
In Orange we've been fighting the past weeks with memory usage a lot...
at the moment incredible huge scenes are being rendered, with multiple
layers and all compositing, stressing limits of memory a lot.
I had hoped that less frequently used blocks would be swapped away
nicely, so fragmented memory could survive. Unfortunately (in OSX) the
malloc range is limited to 2 GB only (upped half of address space).
Other OS's have a limit too, but typically larger afaik.
Now here's mmap to the rescue! It has a very nice feature to map to
a virtual (non existing) file, allowing to allocate disk-mapped memory
on the fly. For as long there's real memory it works nearly as fast as
a regular malloc, and when you go to the swap boundary, it knows nicely
what to swap first.
The upcoming commit will use mmap for all large memory blocks, like
the composit stack, render layers, lamp buffers and images. Tested here
on my 1 GB system, and compositing huge images with a total of 2.5 gig
still works acceptable here. :)
http://www.blender.org/bf/memory.jpg
This is a silly composit test, using 64 MB images with a load of nodes.
Check the header print... the (2323.33M) is the mmap disk-cache in use.
BTW: note that is still limited to the virtual address space of 4 GB.
The new call is:
MEM_mapalloc()
Per definition, mmap() returns zero'ed memory, so a calloc isn't required.
For Windows there's no mmap() available, but I'm pretty sure there's an
equivalent. Windows gurus here are invited to insert that here in code! At
the moment it's nicely ifdeffed, so for Windows the mmap defaults to a
regular alloc.
- Button option "Single" in render-layer panel will enable to only render
the currently indicated render-layer. It will also skip compositing.
- Brought back the 'Local View' render. This will only render the visible
objects, but with lights from the original view-layers.
To make the option useful, it also temporal enables 'Single', which has
the a disadvantage that you need to set the correct render-layer.
It is a bit a tricky option though... since its quite invisble and
confusing for people who don't know the feature. This might become either
a button in 3d header, or use a popup requester to confirm, or... will
need to think over!
At least; both options display in render window a text to denote the option.
using 1 line per part rendered. Might go back to 1 line again, but at this
moment I need the logs for debugging.
Same prints are active now for UI rendering. Just temporal :)
keys with IKEY in buttons to not work.
- Crash in opengl while rendering was caused by the fact that scanline
updates are drawn in the main thread, whilst the actual render thread
then can already be doing different stuff.
Especially with many layers & passes it's getting confusing easily :)
Convention now is that scanline render updates only happen while the
thread is looping over scanlines. As soon as it reached the last, no
drawing happens, not even to update the last segment.
This isnt a problen, since any finished tile is drawn again entirely.
Until now, on each mouse/key event preview render restarted with first tile.
It now rememers where it was, and continues rendering.
Also tried to get threaded preview working, but its more work than I can
spend right now. Back to bugs :)
- Improved stats drawing while rendering, it now draws - while preparing
renderdata - each second the amount of verts/faces.
Also while rendering, the amount of finished and total parts are printed.
- Added ESC in loop that generated Group render data
- On deleting Render Layers, the nodes that use them are now checked and
corrected.
- Restored drawing all scanlines in renderwindow... this wasn't the bug!
I noticed still several cases where the Imbuf library was called within a
thread... and that whilst the Imbuf itself isn't threadsafe. Also the
thread lock I added in rendering for loading images actually didn't
work, because then it was still possible both threads were accessing the
MEM_malloc function at same time.
This commit nearly fully replaces ImBuf calls in compositor (giving another
nice speedup btw, the way preview images in Nodes were calculated used
clumsy imbuf scaling code).
I've also centralized the 'mutex' locking for threading, which now only
resides in BLI_threads.h. This is used to secure the last ImBuf calls
I cannot replace, which is loading images and creating mipmaps.
Really hope we get something more stable now!
You now can set a Preview panel in the Image window, to define a sub-rect
of an image to be processed. Works like the preview in 3D Window. Just
press SHIFT+P to get it activated. Very nice speedup!
This is how it works:
- The compositor still uses the scene image size (including % setting) for
Viewer or Composite output size
- If a preview exists, it calculates the cropped rect from its position
in the Image window, and stores that in the Scene render data
- On composite execute, it copies only this part from the 'generator nodes',
right now Images or Render Results. That makes the entire composite tree
only using small rects, so it will execute fast.
- Also the render window will only display the cropped rect, and on F12
only the cropped part is being executed
- On rendering in background mode, the cropping is ignored though.
Usability notes:
- translating or zooming view will automatically invoke a recalculation
- if you zoom in on details, the calculated rect will even become smaller
- only one Imagewindow can have this Preview Panel, to prevent conflicts of
what the cropped area should be. Compositing is on Scene level, not local
per image window. (Note; 3D Previews are local per window!)
- Closing the preview panel will invoke a full-size recalculation
- All passes/layers from rendering are nicely cropped, including Z and
vectors.
The work to make the compositor do cropping was simple, but getting the
Image window displaying correctly and get all events OK was a lot of work...
indeed, we need to refactor Image Window usage once. Sorry for making the
mess even bigger now. :) I've tried not to interfere with UV edit or Paint
though... only when you're in compositing mode the panel will work.
BUG fix:
3D Preview render didn't work when multiple layers were set in the current
scene.
- Composit cache now gets fully freed on a render. Each output socket of a
node stores the entire image... and while render that's a waste of memory
- Sky 'paper' render was using wrong texture coordinates
- Found missing test_break() in ztransp rendering.
- ZTransp render now also delivers Z values and Speed vectors in passes
Note that speed vectors accumulate within a pixel to store the minimum,
so rendering ztransp on top of a non-moving plane won't give speed...
Best results you get is by rendering it in a separate layer.
The Z value stored is the closest visible transparent face in the pixel.
Fixes:
- Render to 'spare page' has been enabled again. Because of the strict
separation of Render and UI, but especially because a 'render result' now
can consist of unlimited images, I've not made this a Render feature.
Instead, the render-window itself stores the 'spare' image... I also
had to change the convention for it a bit.
Now, instead of having two "render buffers" (which was a render feature),
the RenderWindow will store each previous frame on a re-render. This
storing will only start after you've pressed 'Jkey' once, but then always
will happen for as long the rendered image is same size as previously.
For clarity, I've also renamed the window title, to 'previous frame'.
- RenderWindow shows alpha again on Akey
- Display of the Zvalues in ImageWindow has been tweaked. White now denotes
closest, and the color range goes from camera clip-sta to clip-end.
- Bugfix: on splitting/merging/duplicating windows, the 3D Previewrender was
not always freed correctly, potentially causing crashes or memory leaks.
vertex locations, not global coordinates. This ensures consistant
autosmoothing for each frame. Also fixes missing vectorblur for parts.
Nice task for a dev: put autosmooth code in end of modifier stack... then
it also shows in 3D window
- BUG FIX! I noticed the last tile rendered quite slow, and even did not
update scanlines. Found out that the main tiles processor didn't go
to sleep when the last tile was rendered, because it detected a free
possible thread. This caused the main thread to go into a very tight
loop, eating up a lot of cpu and blocking the other thread.
After a couple of experiments with variable blur filters, I tried
a more interesting, and who knows... original approach. :)
First watch results here:
http://www.blender.org/bf/rt0001_0030.avihttp://www.blender.org/bf/hand0001_0060.avi
These are the steps in producing such results:
- In preprocess, the speed vectors to previous and next frame are
calculated. Speed vectors are screen-aligned and in pixel size.
- while rendering, these vectors get calculated per sample, and
accumulated in the vector buffer checking for "minimum speed".
(on start the vector buffer is initialized on max speed).
- After render:
- The entire image, all pixels, then is converted to quad polygons.
- Also the z value of the pixels is assigned to the polygons
- The vertices for the quads use averaged speed vectors (of the 4
corner faces), using a 'minimum but non-zero' speed rule.
This minimal speed trick works very well to prevent 'tearing' apart
when multiple faces move in different directions in a pixel, or to
be able to separate moving pixels clearly from non-moving ones
- So, now we have a sort of 'mask' of quad polygons. The previous steps
guaranteed that this mask doesn't have antialias color info, and has
speed vectors that ensure individual parts to move nicely without
tearing effects. The Z allows multiple layers of moving masks.
- Then, in temporal buffer, faces get tagged if they move or not
- These tags then go to an anti-alias routine, which assigns alpha
values to edge faces, based on the method we used in past to antialias
bitmaps (still in our code, check the antialias.c in imbuf!)
- finally, the tag buffer is used to tag which z values of the original
image have to be included (to allow blur go behind stuff).
- OK, now we're ready for accumulating! In a loop, all faces then get
drawn (with zbuffer) with increasing influence of their speed vectors.
The resulting image then is accumulated on top of the original with a
decreasing weighting value.
It sounds all quite complex... but the speed is still encouraging. Above
images have 64 mblur steps, which takes about 1-3 seconds per frame.
Usage notes:
- Make sure the render-layer has passes 'Vector' and 'Z' on.
- add in Compositor the VectorBlur node, and connect the image, Z and
speed to the inputs.
- The node allows to set amount of steps (10 steps = 10 forward, 10 back).
and to set a maximum speed in pixels... to prevent extreme moving things
to blur too wide.
- Scene support in RenderLayers
You now can indicate in Compositor to use RenderLayer(s) from other scenes.
Use the new dropdown menu in the "Render Result" node. It will change the
title of the node to indicate that.
The other Scenes are rendered fully separate, creating own databases (and
octrees) after the current scene was finished. They use their own render
settings, with as exception the render output size (and optional border).
This makes the option an interesting memory saver and speedup.
Also note that the render-results of other scenes are kept in memory while
you work. So, after a render, you can tweak all composit effects.
- Render Stats
Added an 'info string' to stats, printed in renderwindow header. It gives
info now on steps "creating database", "shadow buffers", and "octree".
- Bug fixes
Added redraw event for Image window, when using compositor render.
Text objects were not rendered using background render (probably a bug
since depsgraph was added)
Dropdown buttons in Node editor were not refreshed after usage
Sometimes render window did not open, this due to wrong check for 'esc'.
Removed option that renders view-layers on F12, with mouse in 3d window.
Not only was it confusing, it's now more efficient with the Preview Panel,
which does this nicely.
optionally save a jpg next to it, with compression as set in buttons.
This allows quick previews or download from farms.
Button: next to the 'half' and 'zbuf' options for exr.
- RenderLayers with 'view layers' set, now also take visible lights into
account. Works just like for scene layer settings.
- On ESC from render, compositing (if set) is being skipped too
- While rendering with multiple RenderLayers it will end with a display
of the current RenderLayer (as in Scene buttons)
- Enabled Groups to execute in Compositor. They were ignored still.
Note; inside of groups nothing is cached, so a change of a group input
will recalculate it fully. This is needed because groups are linked
data (instances use same internal nodes).
- Made Composit node "Viewer" display correctly input for images with
1/2/3/4 channels.
- Added pass rendering, tested now with only regular Materials. For
Material nodes this is quite more complex... since they cannot be
easily separated in passes (each Material does a full shade)
In this commit all pass render is disabled though, will continue work on
that later.
Sneak preview: http://www.blender.org/bf/rt.jpg (temporal image)
- What did remain is the 'Normal' pass output. Normal works very nice for
relighting effects. Use the "Normal Node" to define where more or less
light should be. (Use "Value Map" node to tweak influence of the
Normal node 'dot' output.)
- EVIL bug fix: I've spend almost a day finding it... when combining AO and
mirror render, the event queue was totally screwing up... two things not
related at all!
Found out error was in ray-mirror code, which was using partially
uninitialized 'ShadeInput' data to pass on to render code.
- Another fix; made sure that while thread render, the threads don't get
events, only the main program will do. Might fix issues reported by
people on linux/windows.
- Live scanline updates while rendering
Using a timer system, each second now the tiles that are being processed
are checked if they could use display.
To make this work pretty, I had to use the threaded 'tile processor' for
a single thread too, but that's now proven to be stable.
Also note that these updates draw per layer, including ztransp progress
separately from solid render.
- Recode of ztransp OSA
Until now (since blender 1.0) the ztransp part was fully rendered and
added on top of the solid part with alpha-over. This adding was done before
the solid part applied sub-pixel sample filtering, causing the ztransp
layer to be always too blurry.
Now the ztransp layer uses same sub=pixel filter, resulting in the same
AA level (and filter results) as the solid part. Quite noticable with hair
renders.
- Vector buffer support & preliminary vector-blur Node
Using the "Render Layer" panel "Vector" pass button, the motion vectors
per pixel are calculated and stored. Accessible via the Compositor.
The vector-blur node is horrible btw! It just uses the length of the
vector to apply a filter like with current (z)blur. I'm committing it anyway,
I'll experiment with it further, and who knows some surprise code shows up!
for compositing code.
Officially malloc/calloc/free is threadsafe, but our secure malloc system
requires all memory blocks to be stored in a single list, so when two
threads write in this list you get conflicts.
