Issue was caused by the cache limitor which was removing 4k textures from the
memory when accessing other images.
This is pretty much awful situation and solved by making it so only image sequences
and movies ace cache-guarded.
Could be optimized further so images used by viewport are not being freed, but
that's much more tricky to do..
This is a nice candidature for 'a'.
Issue was caused by cache limitor removing viewer image buffer
from the memory during compositing. Now made it so all viewer
images are persistent in the memory.
This solves the crash mentioned above and also makes it so
render/compo results are never lost.
Further tweaks are possible, but pretty much happy now, at
least no stoppers for work are there.
This callback is used when cache limiter needs to remove
some cached objects when running out of limit.
From blender side it's used to keep painted images always
in memory.
This fixes issue when painted images were removing from
the memory after image cache rewrite.
Summary:
Behaves very much the same as cache for Movie Clip datablock:
- Image now have `MovieCache *cache` field which replaced
legacy `ListBase ibufs`.
This allows image datablock to easily keep of image
buffers which are owned by itself. This field isn't
saved to the file and getting restored on undo steps.
However, cache limit is global for movies, sequences
and image datablocks now. So overall cached image buffers
size will not go above cache limit size in user
preferences.
- Image buffers which are marked as BITMAPDIRTY will never
be freed from the cache.
- Added utility function to iterate over image buffers
saved in movie cache.
- Movie cache cleanup check callback now have ImBuf argument
which can be used in a condition of cleanup.
- Added some utility functions which replaces legacy ibufs
iterations with image cache iteration which happens from
inside a lock.
- Fixed `image_mem_size()` which was only counting one of
the buffers if both float and byte buffer present.
Additional notes:
- `BKE_image_get_first_ibuf()` is rather stupid, but direct
access to ibufs->first was also the same stupid idea.
Would consider avoid this function is another project.
- There are some places which doesn't look threadsafe, but
they already were not so much threadsafe anyway before.
So think not a big deal with solving this later.
Finally solves infinite memory usage by image sequences! :)
Reviewers: brecht, campbellbarton
Reviewed By: brecht
CC: sebastian_k
Differential Revision: http://developer.blender.org/D95
This commit basically implements frames prefetching for
movie clip datablock.
Number of frames to be prefetched is controlled in User
Preferences, System tab, Prefetch Frames option.
Currently prefetching is destructive-less for movie cache,
meaning mo frames will be removed from the cache when while
prefetching. This is because it's half of simplier to
implement, but it also makes sense from tracking point of
view -- we could want to playback in both directions and
removing frames from behind time cursor is not always a
good idea.
Anyway, smarter prefetching strategy could be developed
later.
Some implementation notes:
- Added MEM_CacheLimiter_get_memory_in_use function to get
memory usage of specified memory limiter.
- Fixed prototype of MEM_CacheLimiter_get_maximum which
was simply wrong (used wrong data type for output).
- Added some utility functions to movie clip and movie
cache for direct cache interaction and obtaining cache
statistics.
- Prefetching is implemented using general jobs system.
which is invoking from clip draw function.
- Prefetcing will stop as soon other job or playback starts.
This is done from performance point of view. Jobs will
likely require lots of CPU power and better to provide
whole CPU to it.
Playback is a bit more complicated case. For jpeg sequence
playback prefetching while paying back is nice. But trying
to prefetch heavy exr images and doing color space
conversion slows down both playback and prefetching.
TODO:
- Think of better policy of dealing with already cached frames
(like when cached frames from other clips prevents frames
from current clip to be prefetched)
- Currently a bit funky redraw notification happens from
prefetch job. Perhaps own ND_ is better to have here.
- Hiding clip while prefetch is active in theory shall stop
prefetching job.
- Having multiple clips opened on file load will prefetch
frames for only one of them.
Memory limitor's queue could be affected when it's being iterated
on enforcing limits -- that's because iteration could free color
managed image buffers.
Fixed by getting least priority element after every element was
freed. Could be optimized a bit, but it anyway shouldn't be so
slow due to specific of cache limiting and limit enforcing finish
condition.
Movie cache is using global memory limitor, which isn't thread safe
in some of operations, so it required to add mutex around limitor
operations in movie cache.
It's probably could be solved in a way with less locks involved by
using different limitor for different areas (like use own limitor
for clips, own limitor for sequencer and so), but that wouldn't be
so easy to control overall memory usage.
--
svn merge -r50125:50126 ^/branches/soc-2011-tomato
Replace pseudo-LRU approach of determining which buffer
to remove when running out of space allowed for cache
with approach which would remove the frame which is most
far away from newly added frame.
This is still a bit tricky because it's impossible to
distinguish which frame to delete in situation of:
CCCC...CC
^
it's either user wants to extend left segment of cached
frames and buffers from right segment should be removed
or he wants to join this two segments and in that case
buffers from right segment should be removed.
Would need a bit more investigation which situation
is more common in general usecase.
Additional changes:
- Cleanup some memutil files (which are familiar to cache limiter)
- Add option to make moviecache verbose. If DEBUG_MESSAGES is
defined in moviecache.c detailed logs would be printed to the
console.
- Movie caches are now named which helps reading debug messages.
