Particle point cache can now be loaded from external files.
- Activated by "external" checkbox in cache panel and giving proper folder path and file name identifier.
- External cache panel has controls for particle emission start, end, lifetime and random lifetime. These should be set according to the external data for correct playback.
- External files should be named "identifier_frame_index.bphys" or "identifier_frame.bphys" where:
* "identifier" is a freely choseable name.
* "frame" is the cached frame number.
** Six digits padded with zeros!, for example "000024".
* "index" can be used to tell caches with the same identifier apart.
** Two digits starting from zero.
** The index and the underscore before are optional. If no index is present the index number in ui should be set to -1.
- Cache file format is pure floating point numbers (in binary, not text!) with each particle's data one after the other with the following data members:
* 3 floats: particle's location vector
* 3 floats: particle's velocity vector (per second)
* 4 floats: particle's rotation quaternion
* 3 floats: particle's angular velocity vector (per second)
* 1 float: frame of the actual data (this can be non-integer for particles that are born or die between two integer frames, but otherwise should be the same as the "frame" in the file name)
- Cache files don't have to exist for each frame.
* Frames without actual data are interpolated from surrounding frames that have data (extrapolation is not supported).
- Cache file formats with extended (or reduced even) data members are in future plans for easier usage.
- Current code only does particles, don't yet know if it's applicable to cloth or sb.
- Known issue: endianness can't yet be handled in any way.
Other changes:
New hard limits for many particle parameters. Some examples:
- Maximum amount of particles: 10M particles :) And before you all go and crash your Blender trying this out remember that this limit is only for those freaks who really have the machine power to handle it. 10M particles alone take around 2.2 Gb of memory / disk space in saved file and each cached frame takes around 0.5 Gb of memory / disk space depending on cache mode.
* Known issue: To actually use this many particles they most likely need to be allocated in parts as taking hold of a 2.2Gb chunk of memory at once is probably not ok with any operating system.
- Maximum amount of children: 100k children/particle (1T childparticles here we come :D)
- Kink frequency: -100k to 100k half-rotations (really strange the previous limit was only from zero upwards)
- Path draw steps: 10 (power of 2 remember)
- Path render steps: 20 (power of 2 also!! If over 1M segments doesn't get you smooth paths then I think nothing will!)
- BLI_add_slash returns the new string length.
- BLI_where_am_i() would often have /./ in the path (not incorrect but annoying, got into python exceptions)
- release/ui/space_image.py, py error referencing invalid keyword args.
- Keyed targets in one list instead of "chaining", this opens up many more possibilities than before and is much less obscure.
- Better keyed timing possibilities (time & duration in frames).
- Looping over keyed targets list.
Other changes:
- New child setting "length" with threshold (great for guard & underfur with a single particle system)
- Modularization of path interpolation code.
- Cleared "animateable" flags from many particle settings that shouldn't be animateable.
Fixes:
- Keyed particles weren't copied properly (ancient bug).
- Hair rotations depended on global z-axis for root rotation so downward facing strands could flip rotation randomly. Now initial hair rotation is derived from face dependent hair matrix. (This caused for example ugly flipping of child strands on some cases).
- Children from faces weren't calculated straight after activating them.
- Multiple disk cache fixes:
* Disk cache didn't work correctly with frame steps.
* Conversion from memory cache to disk cache didn't work with cloth.
* Disk cache crashed on some frames trying to close an already closed cache file.
* Trails didn't work with disk cached particles.
- Child rough effects were effected by emitter object loc/rot making them next to useless with animation, why didn't anybody tell me this!!
- Lots of random code cleanup.
Monthly cleaning round to make it compile warning free.
Mostly it was const stuff (strings, Context), but also
a couple useful fixes, like wrong use of temp pointers.
Only Mathutils callback struct I left alone... design issue.
* Baked normal particles can now use the "Path" visualization.
* Path "max length" & "abs length" are now history:
- New option to set path start & end times + random variation to length.
- Much more flexible (and calculated better) than previous options.
- This works with parents, children, hair & normal particles unlike old length option.
- Only known issue for now is that children from faces don't get calculated correctly when using path start time.
* New option "trails" for "halo", "line" and "billboard" visualizations:
- Draws user controllable number of particle instances along particles path backwards from current position.
- Works with children too for cool/weird visualizations that weren't possible before.
* Normal particle children's velocities are now approximated better when needed so that "line" visualization trails will look nice.
* New particle instance modifier options:
- "path"-option works better and has controllable (max)position along path (with random variation possible).
- "keep shape"-option for hair, keyed, or baked particles allows to place the instances to a single point (with random variation possible) along particle path.
- "axis" option to make rotation handling better (still not perfect, but will have to do for now).
Some fixes & cleanup done along the way:
* Random path length didn't work for non-child particles.
* Cached & unborn particles weren't reset to emit locations.
* Particle numbers weren't drawn in the correct place.
* Setting proper render & draw visualizations was lost somewhere when initializing new particle settings.
* Changing child mode wasn't working correctly.
* Some cleanup & modularization of particle child effector code and particle drawing & rendering code.
* Object & group visualizations didn't work.
* Child simplification didn't work.
* Based on what happens during simulation the cache is marked (also in cache panel, this could possibly be extended to 3d view as well) as:
- exact (not marked)
- outdated (simulation is not done completely with current settings)
- non-exact (frames were skipped during simulation)
* The parameter "cache step" effects the number of frames between saved cache frames.
- This can save a lot of memory (or disk space) if absolutely frame accurate simulation is not required.
- Speeds up the "quick caching" very much.
- Frames between cached frames are interpolated from the cached frames.
- Current default value of 10 frames works nicely with up/down-arrows (skip 10 frames forwards/backwards on timeline), but can be changed if wanted.
* The caching can work in normal or "quick" mode:
[Normal cache]
- Basic: Calculate what even happens (settings change, big frame steps etc.) and cache results, if possible try to use "cache step" when saving cache frames.
- Becomes non-exact: After larger than 1 frame steps.
- Becomes outdated: After any change effecting the simulation other than frame steps.
- Pros/cons: Freedom of doing anything and playing with particles, but exact results have to calculated from the beginning.
[Quick cache]
- Basic: Calculate simulation up to current frame automatically on changes with cache step sized jumps in simulation. With multiple "quick cached" simulations the smallest cache step is used.
- Becomes non-exact: Always from frame 1 (unless cache step = 1).
- Becomes outdated: Never.
- Pros/cons: Not very accurate, but super fast!
- Todo: Transform of any animated (non-autokeyed) object is locked! Probably needs some tinkering with anim sys overrides.
* The simulation can be run forwards or backwards even if it's cache is outdated or non-exact, the following rules apply in these situations:
- step forwards (to unknown) -> simulate from last exact frame, store result
- step backwards (to known) -> result is interpolated from existing frames, store result, clear cache forwards if current frame is after last exact frame
* "Calculate to current frame" runs the simulation from start to current frame with a frame steps of 1.
- Baking does the same, but runs the simulation all the way to the end of simulation.
- Rendering does this automatically if the simulation is outdated of non-exact, so all rendered simulations will always be updated and exact.
* Every cache panel also holds buttons to "Bake all dynamics", "Free all dynamics" and "Update all dynamics to current frame".
* Cloth simulation supports the new cache too.
Mostly harmless ones, except for one about "gzopen64" being
undeclared. This needs some defines in BLI_storage.h to be set
before <unistd.h> is included. Might fix a crash in compressed
file reading, though I'm not sure since it's hard to repeat
the crash consistently.
* Particles support larger than 1 frame changes, bigger frame changes can result in inaccurate results, but it's super fast and you get a nice feeling of how the particles behave!
* "Cache to current frame" button calculates the exact result of particles at current frame.
* Current state of cache can be protected by making it a bake.
* Cache is now in memory by default, disk cache is an option.
* Only "viewport %" number of particles are calculated and cached in viewport, baking and rendering calculate all particles.
* Info on cached frames and memory usage given in ui.
* Support for exact "autocaching" of changes and large frame changes(disabled for now until exact place in event system is decided)
* "Continue physics" is probably deprecated after this and should be removed once sb & cloth use the new cache code.
Todo:
* Make softbody & cloth use the new cache things.
Other changes:
* Some cleanup of particle buttons.
-Many things not yet working properly :)
-Experimental new (hopefully more logical) categorization of buttons, feedback on this is very welcome.
-Separate render & draw types, for example now there is actually a render option "halo" instead of all the different point draw types.
-Particles get recalculated only from buttons that actually change something that has to be recalculated, for example changing visualization doesn't reset particles any more.
-Boid physics buttons are still missing as I'm currently redoing the whole boids code.
-Point caching is still very wip, so baking is not possible for example, but I added a few cache baking flags for rna that were/will be needed for particle buttons logic.
Finally, here is the basic (functional) prototype of the new animation system which will allow for the infamous "everything is animatable", and which also addresses several of the more serious shortcomings of the old system. Unfortunately, this will break old animation files (especially right now, as I haven't written the version patching code yet), however, this is for the future.
Highlights of the new system:
* Scrapped IPO-Curves/IPO/(Action+Constraint-Channels)/Action system, and replaced it with F-Curve/Action.
- F-Curves (animators from other packages will feel at home with this name) replace IPO-Curves.
- The 'new' Actions, act as the containers for F-Curves, so that they can be reused. They are therefore more akin to the old 'IPO' blocks, except they do not have the blocktype restriction, so you can store materials/texture/geometry F-Curves in the same Action as Object transforms, etc.
* F-Curves use RNA-paths for Data Access, hence allowing "every" (where sensible/editable that is) user-accessible setting from RNA to be animated.
* Drivers are no longer mixed with Animation Data, so rigs will not be that easily broken and several dependency problems can be eliminated. (NOTE: drivers haven't been hooked up yet, but the code is in place)
* F-Curve modifier system allows useful 'large-scale' manipulation of F-Curve values, including (I've only included implemented ones here): envelope deform (similar to lattices to allow broad-scale reshaping of curves), curve generator (polynomial or py-expression), cycles (replacing the old cyclic extrapolation modes, giving more control over this). (NOTE: currently this cannot be tested, as there's not access to them, but the code is all in place)
* NLA system with 'tracks' (i.e. layers), and multiple strips per track. (NOTE: NLA system is not yet functional, as it's only partially coded still)
There are more nice things that I will be preparing some nice docs for soon, but for now, check for more details:
http://lists.blender.org/pipermail/bf-taskforce25/2009-January/000260.html
So, what currently works:
* I've implemented two basic operators for the 3D-view only to Insert and Delete Keyframes. These are tempolary ones only that will be replaced in due course with 'proper' code.
* Object Loc/Rot/Scale can be keyframed. Also, the colour of the 'active' material (Note: this should really be for nth material instead, but that doesn't work yet in RNA) can also be keyframed into the same datablock.
* Standard animation refresh (i.e. animation resulting from NLA and Action evaluation) is now done completely separate from drivers before anything else is done after a frame change. Drivers are handled after this in a separate pass, as dictated by depsgraph flags, etc.
Notes:
* Drivers haven't been hooked up yet
* Only objects and data directly linked to objects can be animated.
* Depsgraph will need further tweaks. Currently, I've only made sure that it will update some things in the most basic cases (i.e. frame change).
* Animation Editors are currently broken (in terms of editing stuff). This will be my next target (priority to get Dopesheet working first, then F-Curve editor - i.e. old IPO Editor)
* I've had to put in large chunks of XXX sandboxing for old animation system code all around the place. This will be cleaned up in due course, as some places need special review.
In particular, the particles and sequencer code have far too many manual calls to calculate + flush animation info, which is really bad (this is a 'please explain yourselves' call to Physics coders!).
Think global, act local!
The old favorite G.scene gone! Man... that took almost 2 days.
Also removed G.curscreen and G.edbo.
Not everything could get solved; here's some notes.
- modifiers now store current scene in ModifierData. This is not
meant for permanent, but it can probably stick there until we
cleaned the anim system and depsgraph to cope better with
timing issues.
- Game engine G.scene should become an argument for staring it.
Didn't solve this yet.
- Texture nodes should get scene cfra, but the current implementation
is too tightly wrapped to do it easily.
svn merge https://svn.blender.org/svnroot/bf-blender/trunk/blender -r12987:17416
Issues:
* GHOST/X11 had conflicting changes. Some code was added in 2.5, which was
later added in trunk also, but reverted partially, specifically revision
16683. I have left out this reversion in the 2.5 branch since I think it is
needed there.
http://projects.blender.org/plugins/scmsvn/viewcvs.php?view=rev&root=bf-blender&revision=16683
* Scons had various conflicting changes, I decided to go with trunk version
for everything except priorities and some library renaming.
* In creator.c, there were various fixes and fixes for fixes related to the -w
-W and -p options. In 2.5 -w and -W is not coded yet, and -p is done
differently. Since this is changed so much, and I don't think those fixes
would be needed in 2.5, I've left them out.
* Also in creator.c: there was code for a python bugfix where the screen was not
initialized when running with -P. The code that initializes the screen there
I had to disable, that can't work in 2.5 anymore but left it commented as a
reminder.
Further I had to disable some new function calls. using src/ and python/, as
was done already in this branch, disabled function calls:
* bpath.c: error reporting
* BME_conversions.c: editmesh conversion functions.
* SHD_dynamic: disabled almost completely, there is no python/.
* KX_PythonInit.cpp and Ketsji/ build files: Mathutils is not there, disabled.
* text.c: clipboard copy call.
* object.c: OB_SUPPORT_MATERIAL.
* DerivedMesh.c and subsurf_ccg, stipple_quarttone.
Still to be done:
* Go over files and functions that were moved to a different location but could
still use changes that were done in trunk.
removed frame numbering from BLI_convertstringcode into its own function (BLI_convertstringframe), many uses of BLI_convertstringcode were passing dummy frames values anyway.
in cases where adding the current frame number to a filename is needed run BLI_convertstringframe(...) after BLI_convertstringcode(...)
There are some cases Im not sure BLI_convertstringframe is needed, these have been commented as todo, but at least have the same functionality they used to.
Added BLI_split_dirfile_basic, that only splits the path into directory and file. without checking the dir exists or creating it, without changing the original string that is passed to it.
=======================
Caching and Baking:
- The point cache is now cleared on DAG_object_flush_update(), and not cleared for time dependency graph updates.
- There is now a Bake button instead of Protect. Also cache start and end frames were added to softbody and particles.
- The cloth autoprotect feature was removed.
- The Ctrl+B menu now also bakes cloth and particles next to softbody and fluids. Additionally there are now frree bake and free cache menu entries.
- The point cache api has been changed. There is now a PTCacheID struct for each point cache type that can be filled and then used to call the point cache functions.
- PointCache struct was added to DNA and is automatically allocated for each physics type.
- Soft body now supports Bake Editing just like cloth.
- Tried to make the systems deal consistently with time ipo's and offsets. Still not sure it all works correct, but too complicated to solve completely now.
Library Linking:
- Added some more warnings to prevent editing settings on library linked objects.
- Linked objects now read from the cache located next to the original library file, and never write to it. This restores old behavior for softbodies. For local simulation the mesh and not the object should be linked.
- Dupligroups and proxies can't create local point caches at the moment, how to implement that I'm not sure. We probably need a proxy point cache for that to work (ugh).
Physics UI:
- Renamed deflection panel to collision for consistency and reorganized the buttons. Also removed some softbody collision buttons from the softbody panel that were duplicated in this panel for cloth.
- Tweaked field panel buttons to not jump around when changing options.
- Tabbing e.g. Soft Body Collision into the Soft Body panel, it now only shows Collision to make the panel names readable.
- I tried to make enabled/disabling physics more consistent, since all three system did things different. Now the two modifier buttons to enable the modifier for the viewport and rendering are also duplicated in the physics panels. Toggling the Soft Body and Cloth buttons now both remove their modifiers.
- Fixed modifier error drawing glitch.
Particles:
- Particles are now recalculated more often than before. Previously it did partial updates based on the changes, but that doesn't work well with DAG_object_flush_update() ..
- Fixed memory leak loading keyed particle system. Now keys are not written to file anymore but always created after loading.
- Make particle threads work with autothreads.
Continue Physics:
- The timeline play now has a Continue Physics option in the playback menu, which keeps the simulations going without writing them to the cache.
- This doesn't always work that well, some changes are not immediately updated, but this can be improved later. Still it's fun to get a feel for the physics.
Todo:
- Point cache can get out of sync with and undo and changing a file without saving it.
- Change the point cache file format to store a version (so old point cache files can be either converted or at least ignored), and to do correct endian conversion.
- Menu item and/or buttons for Ctrl+B.
- A system("rm ..") was changed to remove() since the former is very slow for clearing point caches. These system() calls were already giving trouble in a bug in the tracker, but really most use of this system("") should be changed and tested.
- The Soft Body Collision and Clot Collision panel titles don't mention there's point cache settings there too, doing that makes them unreadable with the default panel setup.. but may need to make the names longer anyway.
* Pointcache now uses the process id to construct the path for unsaved files. (so 2 or more blender's open wont try to read/write the same pointcache)
* Temp pointcache is cleared when existing blender, added BIF_clear_tempfiles() for this. Should also be usedto clear EXR's in the temp dir (TODO), BIF_clear_tempfiles also needs to be added in more places. (On file load for instace)
This is messy because the files can be left there when blender quits, however saveBuffers and sessions alredy do :/ Some cleanup function needs to Blender that deals with this!
modifier.c, BKE_modifier.h - flag for modifiers to say they use pointcache, also new func modifiers_usesPointCache
renamed //pointcache to //blendcache_blendfilename so blendfiles in the same dir dont conflict, and other to show this dir isnt limited to pointcache only (nodes way want to use this)
wizard_curve2tree.py - better defaults for pretty tree's
Fixes for bugs in #7833
- Hair softbody cache was reset on rendering.
- When Alt-a playback is synced to audio frames are skipped. This caused dynamics caching to go crazy. Now the first loop of playback goes through all frames if there are non cached dynamic systems.
Invalid point caches weren't cleared when new systems were created. This caused messed up shapes when softbody was added.
=========
Merge of the famous particle patch by Janne Karhu, a full rewrite
of the Blender particle system. This includes:
- Emitter, Hair and Reactor particle types.
- Newtonian, Keyed and Boids physics.
- Various particle visualisation and rendering types.
- Vertex group and texture control for various properties.
- Interpolated child particles from parents.
- Hair editing with combing, growing, cutting, .. .
- Explode modifier.
- Harmonic, Magnetic fields, and multiple falloff types.
.. and lots of other things, some more info is here:
http://wiki.blender.org/index.php/BlenderDev/Particles_Rewritehttp://wiki.blender.org/index.php/BlenderDev/Particles_Rewrite_Doc
The new particle system cannot be backwards compatible. Old particle
systems are being converted to the new system, but will require
tweaking to get them looking the same as before.
Point Cache
===========
The new system to replace manual baking, based on automatic caching
on disk. This is currently used by softbodies and the particle system.
See the Cache API section on:
http://wiki.blender.org/index.php/BlenderDev/PhysicsSprint
Documentation
=============
These new features still need good docs for the release logs, help
for this is appreciated.
