For some reason I thought SDL thread handling would be much simpler... but
the migration to posix pthread went very smooth and painless. Less code
even, and I even notice a slight performance increase!
All threading code is still wrapped in blenlib/intern/threads.c
Only real change was making the callback functions to return void pointer,
instead of an int.
The mutex handling is also different... there's no test anymore if a
mutex was initialized, which is a bit confusing. But it appears to run
all fine still. :)
Nathan Letwory has been signalled already to provide the Windows pthread
library and make/scons linking. For MSVC we might need help from someone
else later though.
In Orange we've been fighting the past weeks with memory usage a lot...
at the moment incredible huge scenes are being rendered, with multiple
layers and all compositing, stressing limits of memory a lot.
I had hoped that less frequently used blocks would be swapped away
nicely, so fragmented memory could survive. Unfortunately (in OSX) the
malloc range is limited to 2 GB only (upped half of address space).
Other OS's have a limit too, but typically larger afaik.
Now here's mmap to the rescue! It has a very nice feature to map to
a virtual (non existing) file, allowing to allocate disk-mapped memory
on the fly. For as long there's real memory it works nearly as fast as
a regular malloc, and when you go to the swap boundary, it knows nicely
what to swap first.
The upcoming commit will use mmap for all large memory blocks, like
the composit stack, render layers, lamp buffers and images. Tested here
on my 1 GB system, and compositing huge images with a total of 2.5 gig
still works acceptable here. :)
http://www.blender.org/bf/memory.jpg
This is a silly composit test, using 64 MB images with a load of nodes.
Check the header print... the (2323.33M) is the mmap disk-cache in use.
BTW: note that is still limited to the virtual address space of 4 GB.
The new call is:
MEM_mapalloc()
Per definition, mmap() returns zero'ed memory, so a calloc isn't required.
For Windows there's no mmap() available, but I'm pretty sure there's an
equivalent. Windows gurus here are invited to insert that here in code! At
the moment it's nicely ifdeffed, so for Windows the mmap defaults to a
regular alloc.
I noticed still several cases where the Imbuf library was called within a
thread... and that whilst the Imbuf itself isn't threadsafe. Also the
thread lock I added in rendering for loading images actually didn't
work, because then it was still possible both threads were accessing the
MEM_malloc function at same time.
This commit nearly fully replaces ImBuf calls in compositor (giving another
nice speedup btw, the way preview images in Nodes were calculated used
clumsy imbuf scaling code).
I've also centralized the 'mutex' locking for threading, which now only
resides in BLI_threads.h. This is used to secure the last ImBuf calls
I cannot replace, which is loading images and creating mipmaps.
Really hope we get something more stable now!
for compositing code.
Officially malloc/calloc/free is threadsafe, but our secure malloc system
requires all memory blocks to be stored in a single list, so when two
threads write in this list you get conflicts.
- Compositor now is threaded
Enable it with the Scene buttons "Threads". This will handle over nodes to
individual threads to be calculated. However, if nodes depend on others
they have to wait. The current system only threads per entire node, not for
calculating results in parts.
I've reshuffled the node execution code to evaluate 'changed' events, and
prepare the entire tree to become simply parsed for open jobs with a call
to node = getExecutableNode()
By default, even without 'thread' option active, all node execution is
done within a separate thread.
Also fixed issues in yesterdays commit for 'event based' calculations, it
didn't do animated images, or execute (on rendering) the correct nodes
when you don't have Render-Result nodes included.
- Added generic Thread support in blenlib/ module
The renderer and the node system now both use same code for controlling the
threads. This has been moved to a new C file in blenlib/intern/threads.c.
Check this c file for an extensive doc and example how to use it.
The current implementation for Compositing allows unlimited amount of
threads. For rendering it is still tied to two threads, although it is
pretty easy to extend to 4 already. People with giant amounts of cpus can
poke me once for tests. :)
- Bugfix in creating group nodes
Group node definitions demand a clear separation of 'internal sockets' and
'external sockets'. The first are sockets being linked internally, the latter
are sockets exposed as sockets for the group itself.
When sockets were linked both internal and external, Blender crashed. It is
solved now by removing the external link(s).
