In Orange we've been fighting the past weeks with memory usage a lot...
at the moment incredible huge scenes are being rendered, with multiple
layers and all compositing, stressing limits of memory a lot.
I had hoped that less frequently used blocks would be swapped away
nicely, so fragmented memory could survive. Unfortunately (in OSX) the
malloc range is limited to 2 GB only (upped half of address space).
Other OS's have a limit too, but typically larger afaik.
Now here's mmap to the rescue! It has a very nice feature to map to
a virtual (non existing) file, allowing to allocate disk-mapped memory
on the fly. For as long there's real memory it works nearly as fast as
a regular malloc, and when you go to the swap boundary, it knows nicely
what to swap first.
The upcoming commit will use mmap for all large memory blocks, like
the composit stack, render layers, lamp buffers and images. Tested here
on my 1 GB system, and compositing huge images with a total of 2.5 gig
still works acceptable here. :)
http://www.blender.org/bf/memory.jpg
This is a silly composit test, using 64 MB images with a load of nodes.
Check the header print... the (2323.33M) is the mmap disk-cache in use.
BTW: note that is still limited to the virtual address space of 4 GB.
The new call is:
MEM_mapalloc()
Per definition, mmap() returns zero'ed memory, so a calloc isn't required.
For Windows there's no mmap() available, but I'm pretty sure there's an
equivalent. Windows gurus here are invited to insert that here in code! At
the moment it's nicely ifdeffed, so for Windows the mmap defaults to a
regular alloc.
* Add ffmpeg-read support in anim.c and util.c
* Makes ImBufs refcountable. You can now increase an internal refcounter
in ImBufs (using IMB_refImBuf) which is decreased by freeImBuf.
This makes it possible to simply pass ImBuf pointers around in the
sequencer saving a few memcopies.
Until now, the zbuffer was written straight from the internal zbuffer,
which has values that are inverse-proportional (like 1.0/z) which makes
it very hard to use it for postprocess, like zblur or other composit effects
that require Z.
Based on info from ILM, the values stored for Z in exr files is the
actual distance from a camera. I think that's about time to migrate to that
convention!
By default now, after render, the z values are converted to floats. This
saves in exr files now, but not in the Iris Z files. That latter was a
blender-only anyway, so might be not a real hassle to drop. :)
You can see the difference in the image window, but notice the range now
is linear mapped from camera clipstart to clipend.
Note; I just discover that ortho Z values need a different correction...
- EXR now saves and reads Zbuffers correctly
- EXR reading didn't set alpha to 1 yet when no alpha buffer was present
- ImageWindow: the "black point" only checked for the r value... now is OK
- ImageWindow: Curves panal has button "reset"
- ImageWindow: hold LMB drag shows rgba and z values. With SHIFT or CTRL it
applies black/white point whilte dragging too
- ImageWindow: saving file copied the entire buffer... removed that. Also
made the header print clear; this save only saves in own file type.
- Curves: zoom and drag now gets clamped by the Clipping value
- Imbuf: duplicate buffer only copied one quarter of to new buffer
- Reading exr images now goes OK. I've unified the code for reading
'half' and 'float' (was nicely possible!). And removed useless copying
of data around.
- Fixed bug in allocating new rects, like for making mipmaps. flag issues.
- filter code accidentally incremented wrong pointer (crash on mipmap too)
Credits go to Gernot Ziegler, who originally coded EXR support, and to
Austin Benesh for bringing it further. Kent Mein provided a lot of code
for integrating float buffers in Blender imbuf and ImBuf API cleanup,
and provided Make and Scons and static linking.
At this moment; the EXR libraries are a *dependency*, so you cannot get
the Orange branch compiled without having OpenEXR installed. Get the
(precompiled or sources) stuff from www.openexr.com. Current default is
that the headers and lib resides in /user/local/
Several changes/additions/fixes were added:
- EXR code only supported 'half' format (16 bits per channel). I've added
float writing, but for reading it I need tomorrow. :)
- Quite some clumsy copying of data happened in EXR code.
- cleaned up the api calls already a bit, preparing for more advanced
support
- Zbuffers were saved 16 bits, now 32 bits
- automatic adding of .exr extensions went wrong
Imbuf:
- added proper imbuf->flags and imbuf->mall support for float buffers, it
was created for *each* imbuf. :)
- found bugs for float buffers in scaling and flipping. Code there will
need more checks still
- imbuf also needs to be verified to behave properly when no 32 bits
rect exists (for saving for example)
TODO:
- support internal float images for textures, backbuf, AO probes, and
display in Image window
Hope this commit won't screwup syncing with bf-blender... :/
