This repository has been archived on 2023-10-09. You can view files and clone it, but cannot push or open issues or pull requests.
Files
blender-archive/source/blender/blenlib/intern/threads.c

232 lines
5.1 KiB
C
Raw Normal View History

Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
/**
*
* $Id:
*
* ***** BEGIN GPL LICENSE BLOCK *****
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software Foundation,
* Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
*
* The Original Code is Copyright (C) 2006 Blender Foundation
* All rights reserved.
*
* The Original Code is: all of this file.
*
* Contributor(s): none yet.
*
* ***** END GPL LICENSE BLOCK *****
*/
#include <math.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
#include "MEM_guardedalloc.h"
#include "BLI_blenlib.h"
#include "BLI_threads.h"
/* ********** basic thread control API ************
Many thread cases have an X amount of jobs, and only an Y amount of
threads are useful (typically amount of cpus)
This code can be used to start a maximum amount of 'thread slots', which
then can be filled in a loop with an idle timer.
A sample loop can look like this (pseudo c);
ListBase lb;
int maxthreads= 2;
int cont= 1;
BLI_init_threads(&lb, do_something_func, maxthreads);
while(cont) {
if(BLI_available_threads(&lb) && !(escape loop event)) {
// get new job (data pointer)
// tag job 'processed
BLI_insert_thread(&lb, job);
}
else PIL_sleep_ms(50);
// find if a job is ready, this the do_something_func() should write in job somewhere
cont= 0;
for(go over all jobs)
if(job is ready) {
if(job was not removed) {
BLI_remove_thread(&lb, job);
}
}
else cont= 1;
}
// conditions to exit loop
if(if escape loop event) {
if(BLI_available_threadslots(&lb)==maxthreads)
break;
}
}
BLI_end_threads(&lb);
************************************************ */
static pthread_mutex_t _malloc_lock = PTHREAD_MUTEX_INITIALIZER;
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
/* just a max for security reasons */
#define RE_MAX_THREAD 8
typedef struct ThreadSlot {
struct ThreadSlot *next, *prev;
void *(*do_thread)(void *);
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
void *callerdata;
pthread_t pthread;
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
int avail;
} ThreadSlot;
void BLI_init_threads(ListBase *threadbase, void *(*do_thread)(void *), int tot)
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
{
int a;
if(threadbase==NULL)
return;
threadbase->first= threadbase->last= NULL;
if(tot>RE_MAX_THREAD) tot= RE_MAX_THREAD;
else if(tot<1) tot= 1;
for(a=0; a<tot; a++) {
ThreadSlot *tslot= MEM_callocN(sizeof(ThreadSlot), "threadslot");
BLI_addtail(threadbase, tslot);
tslot->do_thread= do_thread;
tslot->avail= 1;
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
}
}
/* amount of available threads */
int BLI_available_threads(ListBase *threadbase)
{
ThreadSlot *tslot;
int counter=0;
for(tslot= threadbase->first; tslot; tslot= tslot->next) {
if(tslot->avail)
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
counter++;
}
return counter;
}
/* returns thread number, for sample patterns or threadsafe tables */
int BLI_available_thread_index(ListBase *threadbase)
{
ThreadSlot *tslot;
int counter=0;
for(tslot= threadbase->first; tslot; tslot= tslot->next, counter++) {
if(tslot->avail)
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
return counter;
}
return 0;
}
void BLI_insert_thread(ListBase *threadbase, void *callerdata)
{
ThreadSlot *tslot;
for(tslot= threadbase->first; tslot; tslot= tslot->next) {
if(tslot->avail) {
tslot->avail= 0;
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
tslot->callerdata= callerdata;
pthread_create(&tslot->pthread, NULL, tslot->do_thread, tslot->callerdata);
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
return;
}
}
printf("ERROR: could not insert thread slot\n");
}
void BLI_remove_thread(ListBase *threadbase, void *callerdata)
{
ThreadSlot *tslot;
for(tslot= threadbase->first; tslot; tslot= tslot->next) {
if(tslot->callerdata==callerdata) {
tslot->callerdata= NULL;
pthread_join(tslot->pthread, NULL);
tslot->avail= 1;
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
}
}
}
void BLI_end_threads(ListBase *threadbase)
{
ThreadSlot *tslot;
for(tslot= threadbase->first; tslot; tslot= tslot->next) {
if(tslot->avail==0) {
pthread_join(tslot->pthread, NULL);
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
}
}
BLI_freelistN(threadbase);
}
void BLI_lock_thread(void)
{
pthread_mutex_lock(&_malloc_lock);
}
void BLI_unlock_thread(void)
{
pthread_mutex_unlock(&_malloc_lock);
}
/* ***************** Thread safe MEM_malloc/calloc/free ************************** */
void *MEM_mallocT(int len, char *name)
{
void *mem;
pthread_mutex_lock(&_malloc_lock);
mem= MEM_mallocN(len, name);
pthread_mutex_unlock(&_malloc_lock);
return mem;
}
void *MEM_callocT(int len, char *name)
{
void *mem;
pthread_mutex_lock(&_malloc_lock);
mem= MEM_callocN(len, name);
pthread_mutex_unlock(&_malloc_lock);
return mem;
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
}
Added new malloc type in our MEM module; using the unix feature 'mmap'. In Orange we've been fighting the past weeks with memory usage a lot... at the moment incredible huge scenes are being rendered, with multiple layers and all compositing, stressing limits of memory a lot. I had hoped that less frequently used blocks would be swapped away nicely, so fragmented memory could survive. Unfortunately (in OSX) the malloc range is limited to 2 GB only (upped half of address space). Other OS's have a limit too, but typically larger afaik. Now here's mmap to the rescue! It has a very nice feature to map to a virtual (non existing) file, allowing to allocate disk-mapped memory on the fly. For as long there's real memory it works nearly as fast as a regular malloc, and when you go to the swap boundary, it knows nicely what to swap first. The upcoming commit will use mmap for all large memory blocks, like the composit stack, render layers, lamp buffers and images. Tested here on my 1 GB system, and compositing huge images with a total of 2.5 gig still works acceptable here. :) http://www.blender.org/bf/memory.jpg This is a silly composit test, using 64 MB images with a load of nodes. Check the header print... the (2323.33M) is the mmap disk-cache in use. BTW: note that is still limited to the virtual address space of 4 GB. The new call is: MEM_mapalloc() Per definition, mmap() returns zero'ed memory, so a calloc isn't required. For Windows there's no mmap() available, but I'm pretty sure there's an equivalent. Windows gurus here are invited to insert that here in code! At the moment it's nicely ifdeffed, so for Windows the mmap defaults to a regular alloc.
2006-02-16 17:51:01 +00:00
void *MEM_mapallocT(int len, char *name)
{
void *mem;
pthread_mutex_lock(&_malloc_lock);
Added new malloc type in our MEM module; using the unix feature 'mmap'. In Orange we've been fighting the past weeks with memory usage a lot... at the moment incredible huge scenes are being rendered, with multiple layers and all compositing, stressing limits of memory a lot. I had hoped that less frequently used blocks would be swapped away nicely, so fragmented memory could survive. Unfortunately (in OSX) the malloc range is limited to 2 GB only (upped half of address space). Other OS's have a limit too, but typically larger afaik. Now here's mmap to the rescue! It has a very nice feature to map to a virtual (non existing) file, allowing to allocate disk-mapped memory on the fly. For as long there's real memory it works nearly as fast as a regular malloc, and when you go to the swap boundary, it knows nicely what to swap first. The upcoming commit will use mmap for all large memory blocks, like the composit stack, render layers, lamp buffers and images. Tested here on my 1 GB system, and compositing huge images with a total of 2.5 gig still works acceptable here. :) http://www.blender.org/bf/memory.jpg This is a silly composit test, using 64 MB images with a load of nodes. Check the header print... the (2323.33M) is the mmap disk-cache in use. BTW: note that is still limited to the virtual address space of 4 GB. The new call is: MEM_mapalloc() Per definition, mmap() returns zero'ed memory, so a calloc isn't required. For Windows there's no mmap() available, but I'm pretty sure there's an equivalent. Windows gurus here are invited to insert that here in code! At the moment it's nicely ifdeffed, so for Windows the mmap defaults to a regular alloc.
2006-02-16 17:51:01 +00:00
mem= MEM_mapallocN(len, name);
pthread_mutex_unlock(&_malloc_lock);
Added new malloc type in our MEM module; using the unix feature 'mmap'. In Orange we've been fighting the past weeks with memory usage a lot... at the moment incredible huge scenes are being rendered, with multiple layers and all compositing, stressing limits of memory a lot. I had hoped that less frequently used blocks would be swapped away nicely, so fragmented memory could survive. Unfortunately (in OSX) the malloc range is limited to 2 GB only (upped half of address space). Other OS's have a limit too, but typically larger afaik. Now here's mmap to the rescue! It has a very nice feature to map to a virtual (non existing) file, allowing to allocate disk-mapped memory on the fly. For as long there's real memory it works nearly as fast as a regular malloc, and when you go to the swap boundary, it knows nicely what to swap first. The upcoming commit will use mmap for all large memory blocks, like the composit stack, render layers, lamp buffers and images. Tested here on my 1 GB system, and compositing huge images with a total of 2.5 gig still works acceptable here. :) http://www.blender.org/bf/memory.jpg This is a silly composit test, using 64 MB images with a load of nodes. Check the header print... the (2323.33M) is the mmap disk-cache in use. BTW: note that is still limited to the virtual address space of 4 GB. The new call is: MEM_mapalloc() Per definition, mmap() returns zero'ed memory, so a calloc isn't required. For Windows there's no mmap() available, but I'm pretty sure there's an equivalent. Windows gurus here are invited to insert that here in code! At the moment it's nicely ifdeffed, so for Windows the mmap defaults to a regular alloc.
2006-02-16 17:51:01 +00:00
return mem;
}
void MEM_freeT(void *poin)
{
pthread_mutex_lock(&_malloc_lock);
MEM_freeN(poin);
pthread_mutex_unlock(&_malloc_lock);
}
Three-in-one commit: - Compositor now is threaded Enable it with the Scene buttons "Threads". This will handle over nodes to individual threads to be calculated. However, if nodes depend on others they have to wait. The current system only threads per entire node, not for calculating results in parts. I've reshuffled the node execution code to evaluate 'changed' events, and prepare the entire tree to become simply parsed for open jobs with a call to node = getExecutableNode() By default, even without 'thread' option active, all node execution is done within a separate thread. Also fixed issues in yesterdays commit for 'event based' calculations, it didn't do animated images, or execute (on rendering) the correct nodes when you don't have Render-Result nodes included. - Added generic Thread support in blenlib/ module The renderer and the node system now both use same code for controlling the threads. This has been moved to a new C file in blenlib/intern/threads.c. Check this c file for an extensive doc and example how to use it. The current implementation for Compositing allows unlimited amount of threads. For rendering it is still tied to two threads, although it is pretty easy to extend to 4 already. People with giant amounts of cpus can poke me once for tests. :) - Bugfix in creating group nodes Group node definitions demand a clear separation of 'internal sockets' and 'external sockets'. The first are sockets being linked internally, the latter are sockets exposed as sockets for the group itself. When sockets were linked both internal and external, Blender crashed. It is solved now by removing the external link(s).
2006-01-29 11:36:33 +00:00
/* eof */