This reduces point cache sizes dramatically especially if particle
life time is small compared to total simulation length. For example
with the settings: particle amount = 10000, start = 1, end = 200,
life = 10, cache step = 1, the unoptimized blend file size (compressed)
was a little over 22 Mb and with this optimization the file is a little
under 2 Mb (again compressed). In addition to saving memory/disk space
this also probably speeds up reading from cache, since there's less
data to read.
As an additional fix the memory cache size (displayed in cache panel)
is now calculated correctly.
unrelated changes that ended up being more trouble to commit separate...
- removed BLI_split_dirfile(), was nasty, occasionaly modifying the source string, it could create directories and used the $CWD in some cases. was only used in 2 places in filesel.c, if this gives problems can address without bringing back this function.
renamed BLI_split_dirfile_basic --> BLI_split_dirfile
- view3d_operator_needs_opengl was being called for offscreen render when it wasnt needed.
* The exact location of particle birth wasn't cached if cache step was greater than 1, so the interpolation from an unborn particle to a living one wasn't accurate at all.
* Although not ideal, the current solution is to copy the birth location to the previous cached frame so it's always exact. This should work properly for memory & disk cache.
* This fix shouldn't cause any problems, but exhaustive tests of caching are difficult so be sure to let me know if there's strange behavior.
* Smoke got it's own thread loop now and can be esc'ed now
* Apple OpenMP bug workaround until they fix the GCC (couldn't test this one)
* removing some "static" keyword to prevent crashing on frame display
* should also fix problems with wavelet.noise crashing (Bug #20710)
Fluid:
* sleep delay has been reduced to be more responsive
* Fluid progress is displayed in percent using the mouse cursor (like smoke)
* Better (and windows enabled) OpenMP handling (> 2x-5x speed)
* More Volumetric Texture mapping options (heat, etc) <-- Matt if that's not to your liking, just revert that part, it's separate anyway
* Initial velocity taken from particle settings (no more slow starting)
* Option to select compression method (there seem to be a bug in my high compression usage, at least it's been reported to result in exploding smoke - better use low compression for the time being)
It's been tested since a while but as usual please report any (new!) bugs. ;-)
* Convert all code to use new functions.
* Branch maintainers may want to skip this commit, and run this
conversion script instead, if they use a lot of math functions
in new code:
http://www.pasteall.org/9052/python
The goodies:
* Curves can be used as normal dynamic effectors too with
the new "curve" field shape.
* Group visualization has optional duplication counts for
each object in the specified group.
* Object & group visualizations, which are done without
taking the dupliobject's global position into account
(unless the whole group is used). This is much nicer than
the previous behavior, but I added a "Use Global Location"
option for those who want to use it the old way.
* The active particle system's particles are now drawn a
with theme coloured outline instead of pure white.
* Added object aligned velocity factors (buttons categorized
and re-organized too).
Bug fixes:
* Absorption didn't work as the ui toggle button was forgotten.
* Some other force field ui tweaks.
* Crash after adding children and changing trails count.
* Display types "cross" and "axis" crashed.
* Particles weren't drawn with correct coloring.
* Billboards didn't update properly in viewport to camera
location changes.
* Particle rotation wasn't recreated correctly from point cache.
* Changing particles amount crashed sometimes.
* Some files with child hair crashed on loading.
* Compiler warning fixes.
* Adding boids crashed on frame 1;
* Unified scene wide gravity (currently in scene buttons)
instead of each simulation having it's own gravity.
* Weight parameters for all effectors and an effector group
setting.
* Every effector can use noise.
* Most effectors have "shapes" point, plane, surface, every point.
- "Point" is most like the old effectors and uses the
effector location as the effector point.
- "Plane" uses the closest point on effectors local xy-plane
as the effector point.
- "Surface" uses the closest point on an effector object's
surface as the effector point.
- "Every Point" uses every point in a mesh effector object
as an effector point.
- The falloff is calculated from this point, so for example
with "surface" shape and "use only negative z axis" it's
possible to apply force only "inside" the effector object.
* Spherical effector is now renamed as "force" as it's no longer
just spherical.
* New effector parameter "flow", which makes the effector act as
surrounding air velocity, so the resulting force is
proportional to the velocity difference of the point and "air
velocity". For example a wind field with flow=1.0 results in
proper non-accelerating wind.
* New effector fields "turbulence", which creates nice random
flow paths, and "drag", which slows the points down.
* Much improved vortex field.
* Effectors can now effect particle rotation as well as location.
* Use full, or only positive/negative z-axis to apply force
(note. the z-axis is the surface normal in the case of
effector shape "surface")
* New "force field" submenu in add menu, which adds an empty
with the chosen effector (curve object for corve guides).
* Other dynamics should be quite easy to add to the effector
system too if wanted.
* "Unified" doesn't mean that force fields give the exact same results for
particles, softbody & cloth, since their final effect depends on many external
factors, like for example the surface area of the effected faces.
Code changes
* Subversion bump for correct handling of global gravity.
* Separate ui py file for common dynamics stuff.
* Particle settings updating is flushed with it's id through
DAG_id_flush_update(..).
Known issues
* Curve guides don't yet have all ui buttons in place, but they
should work none the less.
* Hair dynamics don't yet respect force fields.
Other changes
* Particle emission defaults now to frames 1-200 with life of 50
frames to fill the whole default timeline.
* Many particles drawing related crashes fixed.
* Sometimes particles didn't update on first frame properly.
* Hair with object/group visualization didn't work properly.
* Memory leaks with PointCacheID lists (Genscher, remember to
free pidlists after use :).
New stuff
- Bending springs for hair dynamics.
Code cleanup & optimization
- Disabled reactor particles temporarily for cleanup, it's a clumsy system that will be replaced with something better.
- Removed child seams, something better will come here too :)
- Normal particle drawing data is now saved between redraws if the particles don't move between redraws.
* For example rotating the 3d view is now realtime even with 1M particles.
- Many random values for particles now come from a lookup table making things much faster.
- Most accessed small point cache functions are now much faster as macros.
- Lot's of general code cleanup.
- Nothing big should have changed so if something doesn't work like it used to it's probably just a typo somewhere :)
- Hair dynamics have their own panel in particle settings with the settings from cloth panel that apply to hair.
- Basic internal friction force to quickly emulate self collisions and volume preservation. (Still very early code, but gives some idea of what's possible).
- Softbody simulation is no longer used for hair.
* Old files with sb dynamics should just load the hair without dynamics so new dynamics can be applied.
* Invasion of particles exceptions in sb code is finally over.
- Collisions with other objects are disabled for now and will be worked out in the future.
Other changes/fixes:
- Particle mode editing flag wasn't saved properly.
- Some old files with edited hair didn't load correctly.
- Disabled delete & specials menu in particle mode for non-hair editing.
- Fixed yet one more cloth & softbody pointcache update issue.
- Disconnect/connect hair now uses only the deformed mesh so it works correctly also for subsurfed emitters.
- Hair editing now updates correctly with a moving emitter.
* Enable cache for high res + new preview
* Bugfix for smoke banding (in cooperation with N_T)
Hint: Work-in-progress regarding collision objects so can be broken, didn't test
Hint2: jahka enabled a general particle panel but
* bake button doesn't work
* step is not supported for cloth
* several other things there ;)
- Some big refresh issues with softbody & cloth point cache usage should now be fixed.
- Removed sticky objects from particles (better stuff will come back when I get to updating reactor particles).
- Some initial easy memory efficiency cleanup for ParticleData struct. The ultimate goal is to get particles less memory hungry -> more particles possible in single scene.
- Wrong path timing clamping caused hair particles to seem disappeared when changing between normal<->hair particles.
- "Calculate to current frame" in cache buttons baked instead of the intended function.
- Boids particle data is now a bit better organized.
Notifiers
---------
Various fixes for wrong use of notifiers, and some new notifiers
to make things a bit more clear and consistent, with two notable
changes:
* Geometry changes are now done with NC_GEOM, rather than
NC_OBJECT|ND_GEOM_, so an object does need to be available.
* Space data now use NC_SPACE|ND_SPACE_*, instead of data
notifiers or even NC_WINDOW in some cases. Note that NC_SPACE
should only be used for notifying about changes in space data,
we don't want to go back to allqueue(REDRAW..).
Depsgraph
---------
The dependency graph now has a different flush call:
DAG_object_flush_update(scene, ob, flag)
is replaced by:
DAG_id_flush_update(id, flag)
It still works basically the same, one difference is that it now
also accepts object data (e.g. Mesh), again to avoid requiring an
Object to be available. Other ID types will simply do nothing at
the moment.
Docs
----
I made some guidelines for how/when to do which kinds of updates
and notifiers. I can't specify totally exact how to make these
decisions, but these are basically the guidelines I use. So, new
and updated docs are here:
http://wiki.blender.org/index.php/BlenderDev/Blender2.5/NotifiersUpdateshttp://wiki.blender.org/index.php/BlenderDev/Blender2.5/DataNotifiers
- Baked point caches for particles, cloth and softbody can now be edited in particle mode.
* This overwrites the old cloth/sb cache editmode editing.
* The type of editable system is chosen from a menu.
* For particles the current particle system and it's current cache are used.
- Currently this only works for caches that are in memory, but some automatic conversion from disk to memory and back can be implemented later.
- All tools from hair editing can't be applied to point caches and are hidden in the tool panel and specials menu. Some functionality like subdividing paths can be later implemented in a slightly different way from how it works for hair.
- Code is not yet optimized for speed, so editing might be slow sometimes.
Known issues:
- Cloth doesn't update properly while in particle mode, due to the way cloth modifier currently works. Daniel can you check on this?
- As "particle mode" is not only for particles any more some other name would be in place?
- Better icons are needed for the path, point, and tip-modes as the current icons from mesh edit mode are quite misleading.
- Direct editing of point velocities is not yet implemented, but will be in the future.
Other changes:
- Hair editing doesn't require a "make editable" button press any more.
- Multiple caches in single particle system disables changing emission properties.
- Unified ui code for all point cache panels.
* Defined in buttons_particle.py and imported for cloth, smoke & softbody.
- Proper disabling of properties in ui after baking point caches. (Daniel could you please make needed disable code for smoke panels as their functionality is not familiar to me.)
- Hair weight brush has been removed. Once hair dynamics is re-implemented I'll code a more useable alternative to the functionality.
Bug fixes:
- Unlinking particle settings crashed.
- Deleting the active object with particles in the scene crashed.
- Softbody didn't write point caches correctly on save.
Also, made the Outliner's horizontal scrollbar work better for keymaps view. It's still using an approximation of the width, but at least you can scroll now.
* cache for low res (deactivating high res for now)
* new way of view3d rendering of smoke (no longer 3 axes) -using 3dtexture now (introduced into gpu/intern)
* introducing LZO and LZMA libs into extern (makefiles missing for now)
* reducing memory usage after simulating for the frame ended (freeing temporary buffers)
* splitting smoke into 2 modifier for the cache-sake (it cannot handle more than 1 cache on the same modifier-index)
* no color on gui anymore
* fixing non-power-of-2 resolutions (hopefully)
* fixing select-deselect of domain drawing bug
* fixing drawobject.c coding style (making Ton happy) ;-)
HINT #1: If scons doesn't work -> cmakefiles are up-to-date, couldn't test scons (but i tried to mantain them, too)
CODERS HINT #1: we really need a way to disable adding all modifiers through "Add Modifiers" dropdown!
WARNING #1: before applying this commit, deactivate your SMOKE DOMAIN in your old files and save them then. You can open them then savely after that.
WARNING #2: File and cache format of smoke can be changed, this is not final!
- HEADER (beginning of each file)
* general header:
+ 8 char: "BPHYSICS"
+ 1 int: simulation type (same as PTCacheID->type)
* custom header (same for sb, particles and cloth, but can be different for new dynamics)
+ 1 int: totpoint (number of points)
+ 1 int: data_types (bit flags for what the stored data is)
- DATA (directly after header)
*totpoint times the data as specified in data_types flags
- simulation type
soft body = 0, particles = 1, cloth = 2
- data types (more can be added easily when needed)
data flag contains
----------------------------------------
index (1<<0) 1 int (index of current point)
location (1<<1) 3 float
velocity (1<<2) 3 float
rotation (1<<3) 4 float (quaternion)
avelocity (1<<4) 3 float (used for particles)
xconst (1<<4) 3 float (used for cloth)
size (1<<5) 1 float
times (1<<6) 3 float (birth, die & lifetime of particle)
boids (1<<7) 1 BoidData
Notes:
- Every frame is not nescessary since data is interpolated for the inbetween frames.
- For now every point is needed for every cached frame, the "index" data type is reserved for future usage.
- For loading external particle caches only "location" data is necessary, other needed values are determined from the given data.
- Non-dynamic data should be written into an info file if external usage is desired.
* Info file is named as normal cache files, but with frame number 0;
* "Non-dynamic" means data such as particle times.
* Written automatically when baking to disk so basically a library of particle simulations should be possible.
- Old disk cache format is supported for reading, so pre 2.5 files shouldn't break. However old style memory cache (added during 2.5 development) is not supported. To keep memory cached simulations convert the cache to disk cache before svn update and save the blend.
- External sb and cloth caches should be perfectly possible, but due to lack of testing these are not yet enabled in ui.
Other changes:
- Multiple point caches per dynamics system.
* In the future these will hopefully be nla editable etc, but for now things are simple and the current (selected) point cache is used.
* Changing the amount of cached points (for example particle count) is allowed, but might not give correct results if multiple caches are present.
- Generalization of point cache baking etc operator & rna code.
- Comb brushing particle hair didn't work smoothly.
Note: Rna access to softbody point cache is through softbody modifier although the point cache is in softbody settings. This is to make it similar to cloth.
Bugfix: Softbody rna was trying to get "ob->soft->softflag" instead of the correct "ob->softflag".
Particle point cache can now be loaded from external files.
- Activated by "external" checkbox in cache panel and giving proper folder path and file name identifier.
- External cache panel has controls for particle emission start, end, lifetime and random lifetime. These should be set according to the external data for correct playback.
- External files should be named "identifier_frame_index.bphys" or "identifier_frame.bphys" where:
* "identifier" is a freely choseable name.
* "frame" is the cached frame number.
** Six digits padded with zeros!, for example "000024".
* "index" can be used to tell caches with the same identifier apart.
** Two digits starting from zero.
** The index and the underscore before are optional. If no index is present the index number in ui should be set to -1.
- Cache file format is pure floating point numbers (in binary, not text!) with each particle's data one after the other with the following data members:
* 3 floats: particle's location vector
* 3 floats: particle's velocity vector (per second)
* 4 floats: particle's rotation quaternion
* 3 floats: particle's angular velocity vector (per second)
* 1 float: frame of the actual data (this can be non-integer for particles that are born or die between two integer frames, but otherwise should be the same as the "frame" in the file name)
- Cache files don't have to exist for each frame.
* Frames without actual data are interpolated from surrounding frames that have data (extrapolation is not supported).
- Cache file formats with extended (or reduced even) data members are in future plans for easier usage.
- Current code only does particles, don't yet know if it's applicable to cloth or sb.
- Known issue: endianness can't yet be handled in any way.
Other changes:
New hard limits for many particle parameters. Some examples:
- Maximum amount of particles: 10M particles :) And before you all go and crash your Blender trying this out remember that this limit is only for those freaks who really have the machine power to handle it. 10M particles alone take around 2.2 Gb of memory / disk space in saved file and each cached frame takes around 0.5 Gb of memory / disk space depending on cache mode.
* Known issue: To actually use this many particles they most likely need to be allocated in parts as taking hold of a 2.2Gb chunk of memory at once is probably not ok with any operating system.
- Maximum amount of children: 100k children/particle (1T childparticles here we come :D)
- Kink frequency: -100k to 100k half-rotations (really strange the previous limit was only from zero upwards)
- Path draw steps: 10 (power of 2 remember)
- Path render steps: 20 (power of 2 also!! If over 1M segments doesn't get you smooth paths then I think nothing will!)
- BLI_add_slash returns the new string length.
- BLI_where_am_i() would often have /./ in the path (not incorrect but annoying, got into python exceptions)
- release/ui/space_image.py, py error referencing invalid keyword args.
- Keyed targets in one list instead of "chaining", this opens up many more possibilities than before and is much less obscure.
- Better keyed timing possibilities (time & duration in frames).
- Looping over keyed targets list.
Other changes:
- New child setting "length" with threshold (great for guard & underfur with a single particle system)
- Modularization of path interpolation code.
- Cleared "animateable" flags from many particle settings that shouldn't be animateable.
Fixes:
- Keyed particles weren't copied properly (ancient bug).
- Hair rotations depended on global z-axis for root rotation so downward facing strands could flip rotation randomly. Now initial hair rotation is derived from face dependent hair matrix. (This caused for example ugly flipping of child strands on some cases).
- Children from faces weren't calculated straight after activating them.
- Multiple disk cache fixes:
* Disk cache didn't work correctly with frame steps.
* Conversion from memory cache to disk cache didn't work with cloth.
* Disk cache crashed on some frames trying to close an already closed cache file.
* Trails didn't work with disk cached particles.
- Child rough effects were effected by emitter object loc/rot making them next to useless with animation, why didn't anybody tell me this!!
- Lots of random code cleanup.
Monthly cleaning round to make it compile warning free.
Mostly it was const stuff (strings, Context), but also
a couple useful fixes, like wrong use of temp pointers.
Only Mathutils callback struct I left alone... design issue.
* Baked normal particles can now use the "Path" visualization.
* Path "max length" & "abs length" are now history:
- New option to set path start & end times + random variation to length.
- Much more flexible (and calculated better) than previous options.
- This works with parents, children, hair & normal particles unlike old length option.
- Only known issue for now is that children from faces don't get calculated correctly when using path start time.
* New option "trails" for "halo", "line" and "billboard" visualizations:
- Draws user controllable number of particle instances along particles path backwards from current position.
- Works with children too for cool/weird visualizations that weren't possible before.
* Normal particle children's velocities are now approximated better when needed so that "line" visualization trails will look nice.
* New particle instance modifier options:
- "path"-option works better and has controllable (max)position along path (with random variation possible).
- "keep shape"-option for hair, keyed, or baked particles allows to place the instances to a single point (with random variation possible) along particle path.
- "axis" option to make rotation handling better (still not perfect, but will have to do for now).
Some fixes & cleanup done along the way:
* Random path length didn't work for non-child particles.
* Cached & unborn particles weren't reset to emit locations.
* Particle numbers weren't drawn in the correct place.
* Setting proper render & draw visualizations was lost somewhere when initializing new particle settings.
* Changing child mode wasn't working correctly.
* Some cleanup & modularization of particle child effector code and particle drawing & rendering code.
* Object & group visualizations didn't work.
* Child simplification didn't work.
* Based on what happens during simulation the cache is marked (also in cache panel, this could possibly be extended to 3d view as well) as:
- exact (not marked)
- outdated (simulation is not done completely with current settings)
- non-exact (frames were skipped during simulation)
* The parameter "cache step" effects the number of frames between saved cache frames.
- This can save a lot of memory (or disk space) if absolutely frame accurate simulation is not required.
- Speeds up the "quick caching" very much.
- Frames between cached frames are interpolated from the cached frames.
- Current default value of 10 frames works nicely with up/down-arrows (skip 10 frames forwards/backwards on timeline), but can be changed if wanted.
* The caching can work in normal or "quick" mode:
[Normal cache]
- Basic: Calculate what even happens (settings change, big frame steps etc.) and cache results, if possible try to use "cache step" when saving cache frames.
- Becomes non-exact: After larger than 1 frame steps.
- Becomes outdated: After any change effecting the simulation other than frame steps.
- Pros/cons: Freedom of doing anything and playing with particles, but exact results have to calculated from the beginning.
[Quick cache]
- Basic: Calculate simulation up to current frame automatically on changes with cache step sized jumps in simulation. With multiple "quick cached" simulations the smallest cache step is used.
- Becomes non-exact: Always from frame 1 (unless cache step = 1).
- Becomes outdated: Never.
- Pros/cons: Not very accurate, but super fast!
- Todo: Transform of any animated (non-autokeyed) object is locked! Probably needs some tinkering with anim sys overrides.
* The simulation can be run forwards or backwards even if it's cache is outdated or non-exact, the following rules apply in these situations:
- step forwards (to unknown) -> simulate from last exact frame, store result
- step backwards (to known) -> result is interpolated from existing frames, store result, clear cache forwards if current frame is after last exact frame
* "Calculate to current frame" runs the simulation from start to current frame with a frame steps of 1.
- Baking does the same, but runs the simulation all the way to the end of simulation.
- Rendering does this automatically if the simulation is outdated of non-exact, so all rendered simulations will always be updated and exact.
* Every cache panel also holds buttons to "Bake all dynamics", "Free all dynamics" and "Update all dynamics to current frame".
* Cloth simulation supports the new cache too.
Mostly harmless ones, except for one about "gzopen64" being
undeclared. This needs some defines in BLI_storage.h to be set
before <unistd.h> is included. Might fix a crash in compressed
file reading, though I'm not sure since it's hard to repeat
the crash consistently.
* Particles support larger than 1 frame changes, bigger frame changes can result in inaccurate results, but it's super fast and you get a nice feeling of how the particles behave!
* "Cache to current frame" button calculates the exact result of particles at current frame.
* Current state of cache can be protected by making it a bake.
* Cache is now in memory by default, disk cache is an option.
* Only "viewport %" number of particles are calculated and cached in viewport, baking and rendering calculate all particles.
* Info on cached frames and memory usage given in ui.
* Support for exact "autocaching" of changes and large frame changes(disabled for now until exact place in event system is decided)
* "Continue physics" is probably deprecated after this and should be removed once sb & cloth use the new cache code.
Todo:
* Make softbody & cloth use the new cache things.
Other changes:
* Some cleanup of particle buttons.
-Many things not yet working properly :)
-Experimental new (hopefully more logical) categorization of buttons, feedback on this is very welcome.
-Separate render & draw types, for example now there is actually a render option "halo" instead of all the different point draw types.
-Particles get recalculated only from buttons that actually change something that has to be recalculated, for example changing visualization doesn't reset particles any more.
-Boid physics buttons are still missing as I'm currently redoing the whole boids code.
-Point caching is still very wip, so baking is not possible for example, but I added a few cache baking flags for rna that were/will be needed for particle buttons logic.
