archive/blender-archive
Archived
1
1
Fork 0
Code Pull Requests Packages Activity

Added new malloc type in our MEM module; using the unix feature 'mmap'.

Browse Source 
In Orange we've been fighting the past weeks with memory usage a lot...
at the moment incredible huge scenes are being rendered, with multiple
layers and all compositing, stressing limits of memory a lot.
I had hoped that less frequently used blocks would be swapped away
nicely, so fragmented memory could survive. Unfortunately (in OSX) the
malloc range is limited to 2 GB only (upped half of address space).
Other OS's have a limit too, but typically larger afaik.

Now here's mmap to the rescue! It has a very nice feature to map to
a virtual (non existing) file, allowing to allocate disk-mapped memory
on the fly. For as long there's real memory it works nearly as fast as
a regular malloc, and when you go to the swap boundary, it knows nicely
what to swap first.

The upcoming commit will use mmap for all large memory blocks, like
the composit stack, render layers, lamp buffers and images. Tested here
on my 1 GB system, and compositing huge images with a total of 2.5 gig
still works acceptable here. :)

http://www.blender.org/bf/memory.jpg
This is a silly composit test, using 64 MB images with a load of nodes.
Check the header print... the (2323.33M) is the mmap disk-cache in use.

BTW: note that is still limited to the virtual address space of 4 GB.

The new call is:
MEM_mapalloc()

Per definition, mmap() returns zero'ed memory, so a calloc isn't required.

For Windows there's no mmap() available, but I'm pretty sure there's an
equivalent. Windows gurus here are invited to insert that here in code! At
the moment it's nicely ifdeffed, so for Windows the mmap defaults to a
regular alloc.
This commit is contained in:
Ton Roosendaal
2006-02-16 17:51:01 +00:00
parent 7f4b01ccf0
commit fe036a0538
11 changed files with 151 additions and 101 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
source/blender/render/intern/source/pipeline.c
View File

@@ -183,12 +183,12 @@ static void render_layer_add_pass(RenderLayer *rl, int rectsize, int passtype, c
int x;
/* initialize to max speed */
rect= rpass->rect= MEM_mallocT(sizeof(float)*rectsize, mallocstr);
rect= rpass->rect= MEM_mapallocT(sizeof(float)*rectsize, mallocstr);
for(x= rectsize-1; x>=0; x--)
rect[x]= PASS_VECTOR_MAX;
}
else
rpass->rect= MEM_callocT(sizeof(float)*rectsize, mallocstr);
rpass->rect= MEM_mapallocT(sizeof(float)*rectsize, mallocstr);
}
float *RE_RenderLayerGetPass(RenderLayer *rl, int passtype)
@@ -244,7 +244,7 @@ static RenderResult *new_render_result(Render *re, rcti *partrct, int crop)
rl->layflag= srl->layflag;
rl->passflag= srl->passflag;
rl->rectf= MEM_callocT(rectx*recty*sizeof(float)*4, "layer float rgba");
rl->rectf= MEM_mapallocT(rectx*recty*sizeof(float)*4, "layer float rgba");
if(srl->passflag & SCE_PASS_Z)
render_layer_add_pass(rl, rectx*recty, SCE_PASS_Z, "Layer float Z");
@@ -271,7 +271,7 @@ static RenderResult *new_render_result(Render *re, rcti *partrct, int crop)
rl= MEM_callocT(sizeof(RenderLayer), "new render layer");
BLI_addtail(&rr->layers, rl);
rl->rectf= MEM_callocT(rectx*recty*sizeof(float)*4, "prev/env float rgba");
rl->rectf= MEM_mapallocT(rectx*recty*sizeof(float)*4, "prev/env float rgba");
/* note, this has to be in sync with scene.c */
rl->lay= (1<<20) -1;
@@ -1159,7 +1159,7 @@ static void do_write_image_or_movie(Render *re, Scene *scene, bMovieHandle *mh)
int dofree = 0;
/* note; the way it gets 32 bits rects is weak... */
if(rres.rect32==NULL) {
rres.rect32= MEM_mallocT(sizeof(int)*rres.rectx*rres.recty, "temp 32 bits rect");
rres.rect32= MEM_mapallocT(sizeof(int)*rres.rectx*rres.recty, "temp 32 bits rect");
dofree = 1;
}
RE_ResultGet32(re, rres.rect32);