A full detailed description of this will be done later... is several days
of work. Here's a summary:
Render:
- Full cleanup of render code, removing *all* globals and bad level calls
all over blender. Render module is now not called abusive anymore
- API-fied calls to rendering
- Full recode of internal render pipeline. Is now rendering tiles by
default, prepared for much smarter 'bucket' render later.
- Each thread now can render a full part
- Renders were tested with 4 threads, goes fine, apart from some lookup
tables in softshadow and AO still
- Rendering is prepared to do multiple layers and passes
- No single 32 bits trick in render code anymore, all 100% floats now.
Writing images/movies
- moved writing images to blender kernel (bye bye 'schrijfplaatje'!)
- made a new Movie handle system, also in kernel. This will enable much
easier use of movies in Blender
PreviewRender:
- Using new render API, previewrender (in buttons) now uses regular render
code to generate images.
- new datafile 'preview.blend.c' has the preview scenes in it
- previews get rendered in exact displayed size (1 pixel = 1 pixel)
3D Preview render
- new; press Pkey in 3d window, for a panel that continuously renders
(pkey is for games, i know... but we dont do that in orange now!)
- this render works nearly identical to buttons-preview render, so it stops
rendering on any event (mouse, keyboard, etc)
- on moving/scaling the panel, the render code doesn't recreate all geometry
- same for shifting/panning view
- all other operations (now) regenerate the full render database still.
- this is WIP... but big fun, especially for simple scenes!
Compositor
- Using same node system as now in use for shaders, you can composit images
- works pretty straightforward... needs much more options/tools and integration
with rendering still
- is not threaded yet, nor is so smart to only recalculate changes... will be
done soon!
- the "Render Result" node will get all layers/passes as output sockets
- The "Output" node renders to a builtin image, which you can view in the Image
window. (yes, output nodes to render-result, and to files, is on the list!)
The Bad News
- "Unified Render" is removed. It might come back in some stage, but this
system should be built from scratch. I can't really understand this code...
I expect it is not much needed, especially with advanced layer/passes
control
- Panorama render, Field render, Motion blur, is not coded yet... (I had to
recode every single feature in render, so...!)
- Lens Flare is also not back... needs total revision, might become composit
effect though (using zbuffer for visibility)
- Part render is gone! (well, thats obvious, its default now).
- The render window is only restored with limited functionality... I am going
to check first the option to render to a Image window, so Blender can become
a true single-window application. :)
For example, the 'Spare render buffer' (jkey) doesnt work.
- Render with border, now default creates a smaller image
- No zbuffers are written yet... on the todo!
- Scons files and MSVC will need work to get compiling again
OK... thats what I can quickly recall. Now go compiling!
causing render to crash on cygwin with gcc 3.4.4. Changed
declaration of shadepixel() argument "int facenr" to volatile.
Can anybody recommend a more elegant solution?
Until now, the zbuffer was written straight from the internal zbuffer,
which has values that are inverse-proportional (like 1.0/z) which makes
it very hard to use it for postprocess, like zblur or other composit effects
that require Z.
Based on info from ILM, the values stored for Z in exr files is the
actual distance from a camera. I think that's about time to migrate to that
convention!
By default now, after render, the z values are converted to floats. This
saves in exr files now, but not in the Iris Z files. That latter was a
blender-only anyway, so might be not a real hassle to drop. :)
You can see the difference in the image window, but notice the range now
is linear mapped from camera clipstart to clipend.
Note; I just discover that ortho Z values need a different correction...
- Image textures use float colors now, when present. Works for mipmap too,
and for AO skycolor (probes)
- Backbuffer option uses float buffers too. Note that rendering OSA will
resample the backbuffer, filtering it... will need to be solved with the
new composit stage
- LMB sampling in image window now shows float color too
+ bugfix in imbuf, filtering for float buffers had an error.
Accidentally didn't disable the tests I did for 'translucent raytrace',
a cheapskate SSS method;
http://www.blender.org/bf/rt.jpghttp://www.blender.org/bf/rt1.jpg
It slowed down render too much, was default on... but invisible! It needed
the F10 buttons "rt" to be huge (600-1000)
Credits go to Gernot Ziegler, who originally coded EXR support, and to
Austin Benesh for bringing it further. Kent Mein provided a lot of code
for integrating float buffers in Blender imbuf and ImBuf API cleanup,
and provided Make and Scons and static linking.
At this moment; the EXR libraries are a *dependency*, so you cannot get
the Orange branch compiled without having OpenEXR installed. Get the
(precompiled or sources) stuff from www.openexr.com. Current default is
that the headers and lib resides in /user/local/
Several changes/additions/fixes were added:
- EXR code only supported 'half' format (16 bits per channel). I've added
float writing, but for reading it I need tomorrow. :)
- Quite some clumsy copying of data happened in EXR code.
- cleaned up the api calls already a bit, preparing for more advanced
support
- Zbuffers were saved 16 bits, now 32 bits
- automatic adding of .exr extensions went wrong
Imbuf:
- added proper imbuf->flags and imbuf->mall support for float buffers, it
was created for *each* imbuf. :)
- found bugs for float buffers in scaling and flipping. Code there will
need more checks still
- imbuf also needs to be verified to behave properly when no 32 bits
rect exists (for saving for example)
TODO:
- support internal float images for textures, backbuf, AO probes, and
display in Image window
Hope this commit won't screwup syncing with bf-blender... :/
Threadsafe patch for environment maps type "Load" missed to include a
call, so still crashed. Only for non-debug builds though, so not reported
earlier.
From my cvs log 7 months ago:
"Added threadsafe patch from Martin.
Now envmaps of type "Load" should not give errors. I assume Martin tested!"
:)
With the fix over a month ago, which added correct texture space vectors for
the bump, gave results so crispy that normals could invert after normalize.
This only when the normal "fac" slider was > 1.0.
The normals from imagetextures now get clipped to prevent it to result in
flipping normals. Will do more tests though...
Also note that the real good way would be have the tangent vectors for the
actual render normal available to perturb for bump, thats another story.
Material Nodes now have input sockets, to override the actual Material
settings. If socket has no input, you can also edit Material settings here.
(Color picker, number slider).
Of course a load of options will become available here, first bugfixing!
(Oh, and enable fix thread render)
the coordinate outputs now have correct dx/dy vectors for Image AA, and
texture delivers correct intensity, rgb, alpha and normal.
Note; we need a "Vector Mapping" node, to do 2d/3d mapping, like in the
Material "Map In" panel.
**** NEW: Group Nodes
Node trees usually become messy and confusing quickly, so we need
not only a way to collapse Nodes into single 'groups', but also a
way to re-use that data to create libraries of effects.
This has been done by making a new Library data type, the NodeTree.
Everything that has been grouped is stored here, and available for
re-use, appending or linking. These NodeTrees are fully generic,
i.e. can store shader trees, composit trees, and so on. The 'type'
value as stored in the NodeTree will keep track of internal type
definitions and execute/drawing callbacks. Needless to say, re-using
shader trees in a composit tree is a bit useless, and will be
prevented in the browsing code. :)
So; any NodeTree can become a "Goup Node" inside in a NodeTree. This
Group Node then works just like any Node.
To prevent the current code to become too complex, I've disabled
the possibility to insert Groups inside of Groups. That might be
enabled later, but is a real nasty piece of code to get OK.
Since Group Nodes are a dynamic Node type, a lot of work has been
done to ensure Node definitions can be dynamic too, but still allow
to be stored in files, and allow to be verified for type-definition
changes on reloading. This system needs a little bit maturing still,
so the Python gurus should better wait a little bit! (Also for me to
write the definite API docs for it).
What works now:
- Press CTRL+G to create a new Group. The grouping code checks for
impossible selections (like an unselected node between selected nodes).
Everthing that's selected then gets removed from the current tree, and
inserted in a new NodeTree library data block. A Group Node then is
added which links to this new NodeTree.
- Press ALT+G to ungroup. This will not delete the NodeTree library
data, but just duplicate the Group into the current tree.
- Press TAB, or click on the NodeTree icon to edit Groups. Note that
NodeTrees are instances, so editing one Group will also change the
other users.
This also means that when removing nodes in a Group (or hiding sockets
or changing internal links) this is immediately corrected for all users
of this Group, also in other Materials.
- While editing Groups, only the internal Nodes can be edited. A single
click outside of the Group boundary will close this 'edit mode'.
What needs to be done:
- SHIFT+A menu in toolbox style, also including a list of Groups
- Enable the single-user button in the Group Node
- Displaying all (visible) internal group UI elements in the Node Panel
- Enable Library linking and prevent editing of Groups then.
**** NEW: Socket Visibility control
Node types will be generated with a lot of possible inputs or outputs,
and drawing all sockets all the time isn't very useful then.
A new option in the Node header ('plus' icon) allows to either hide all
unused sockets (first keypress) or to reveil them (when there are hidden
sockets, the icon displays black, otherwise it's blended).
Hidden sockets in Nodes also are not exported to a Group, so this way
you can control what options (in/outputs) exactly are available.
To be done:
- a way to hide individual sockets, like with a RMB click on it.
**** NEW: Nodes now render!
This is still quite primitive, more on a level to replace the (now
obsolete and disabled) Material Layers.
What needs to be done:
- make the "Geometry" node work properly, also for AA textures
- make the Texture Node work (does very little at the moment)
- give Material Nodes all inputs as needed (like Map-to Panel)
- find a way to export more data from a Material Node, like the
shadow value, or light intensity only, etc
Very important also to separate from the Material Buttons the
"global" options, like "Ztransp" or "Wire" or "Halo". These can not
be set for each Material-Node individually.
Also note that the Preview Render (Buttons window) now renders a bit
differently. This was a horrid piece of antique code, using a totally
incompatible way of rendering. Target is to fully re-use internal
render code for previews.
OK... that's it mostly. Now test!
********* Node editor work:
- To enable Nodes for Materials, you have to set the "Use Nodes"
button, in the new Material buttons "Nodes" Panel or in header
of the Node editor. Doing this will disable Material-Layers.
- Nodes now execute materials ("shaders"), but still only using the
previewrender code.
- Nodes have (optional) previews for rendered images.
- Node headers allow to hide buttons and/or preview image
- Nodes can be dragged larger/smaller (right-bottom corner)
- Nodes can be hidden (minimized) with hotkey H
- CTRL+click on an Input Socket gives a popup with default values.
- Changing Material/Texture or Mix node will adjust Node title.
- Click-drag outside of a Node changes cursor to "Knife' and allows to
draw a rect where to cut Links.
- Added new node types RGBtoBW, Texture, In/Output, ColorRamp
- Material Nodes have options to ouput diffuse or specular, or to use
a negative normal. The input socket 'Normal' will force the material
to use that normal, otherwise it uses the normal from the Material
that has the node tree.
- When drawing a link between two not-matching sockets, Blender inserts
a converting node (now only for value/rgb combos)
- When drawing a link to an input socket that's already in use, the
old link will either disappear or flip to another unused socket.
- A click on a Material Node will activate it, and show all its settings
in the Material Buttons. Active Material Nodes draw the material icon
in red.
- A click on any node will show its options in the Node Panel in the
Material buttons.
- Multiple Output Nodes can be used, to sample contents of a tree, but
only one Output is the real one, which is indicated in a different
color and red material icon.
- Added ThemeColors for node types
- ALT+C will convert existing Material-Layers to Node... this currently
only adds the material/mix nodes and connects them. Dunno if this is
worth a lot of coding work to make perfect?
- Press C to call another "Solve order", which will show all possible
cyclic conflicts (if there are).
- Technical: nodes now use "Type" structs which define the
structure of nodes and in/output sockets. The Type structs store all
fixed info, callbacks, and allow to reconstruct saved Nodes to match
what is required by Blender.
- Defining (new) nodes now is as simple as filling in a fixed
Type struct, plus code some callbacks. A doc will be made!
- Node preview images are by default float
********* Icon drawing:
- Cleanup of how old icons were implemented in new system, making
them 16x16 too, correctly centered *and* scaled.
- Made drawing Icons use float coordinates
- Moved BIF_calcpreview_image() into interface_icons.c, renamed it
icon_from_image(). Removed a lot of unneeded Imbuf magic here! :)
- Skipped scaling and imbuf copying when icons are OK size
********* Preview render:
- Huge cleanup of code....
- renaming BIF_xxx calls that only were used internally
- BIF_previewrender() now accepts an argument for rendering method,
so it supports icons, buttonwindow previewrender and node editor
- Only a single BIF_preview_changed() call now exists, supporting all
signals as needed for buttos and node editor
********* More stuff:
- glutil.c, glaDrawPixelsSafe() and glaDrawPixelsTex() now accept format
argument for GL_FLOAT rects
- Made the ColorBand become a built-in button for interface.c
Was a load of cleanup work in buttons_shading.c...
- removed a load of unneeded glBlendFunc() calls
- Fixed bug in calculating text length for buttons (ancient!)
More bump issues... Env made very nice examples, but all of it quite nasty
to solve.
Three improvements I found nevertheless:
- bug in mipmap code for bump gave errors in one derivate, causing aliasing
- when Image texture option "Clip" was used, the imagetexture function
returned wrong value, causing the caller to create own normal based on
color (zero!).
- Mipmapped bump now also interpolates the samples (never did this...)
Uninitialized 'alpha' flag in Image texture could cause render errors.
Didn't show in OSX though... (stupid system here inits stack to zero).
But, Ken Hughes found this, so all credits are for him!
no shadow a little bit. Uses same threshold function as the "Bias" button.
(That latter still works, and will use an automatic bias value based on the
geometry, to prevent terminator errors in raytracing).
Anyhoo, with this manual bias you can get rid of terminator problems for
weird diffuse shaders now as well (like tangent or fresnel).
Committed a few more files than needed, that's just code cleanup.
Another ancient bug in unified render... the lamphalo wasn't filled in
correctly over sky, especially when the background had halos (or stars).
Only shows when using filters other than box though.
wire frame became very limited... the information of faces (vertex colors
and UV reside there) got lost.
Solved it nicely with creating a large index table, and use bsearch() to
get quickly the matching information.
Then I noticed the render code needed fixes too for wire, no proper UV's
were calculated over the wire edge.
Unified render didn't treat the alpha for raytraced transparent correct...
unified is based on layering with alpha, so it cannot support alpha in
raytraced transparent correctly.
I'm going to kill unified one day... its giving me headaches. :)
-> Any Group Duplicate now can get local timing and local NLA override. This
enables to control the entire animation system of the Group.
Two methods for this have been implemented.
1) The quick way: just give the duplicator a "Startframe" offset.
2) Advanced: in the NLA Editor you can add ActionStrips to the duplicator
to override NLA/action of any Grouped Object.
For "Group NLA" to work, an ActionStrip needs to know which Object in a
group it controls. On adding a strip, the code checks if an Action was
already used by an Object in the Group, and assigns it automatic to that
Object.
You can also set this in the Nkey "Properties" panel for the strip.
Change in NLA: the SHIFT+A "Add strip" command now always adds strips to
the active Object. (It used to check where mouse was). This allows to add
NLA strips to Objects that didn't have actions/nla yet.
Important note: In Blender, duplicates are fully procedural and generated
on the fly for each redraw. This means that redraw speed equals to stepping
through frames, when using animated Duplicated Groups.
-> Recoded entire duplicator system
The old method was antique and clumsy, using globals and full temporal
copies of Object. The new system is nicer in control, faster, and since it
doesn't use temporal object copies anymore, it works better with Derived
Mesh and DisplayList and rendering.
By centralizing the code for duplicating, more options can be easier added.
Features to note:
- Duplicates now draw selected/unselected based on its Duplicator setting.
- Same goes for the drawtype (wire, solid, selection outline, etc)
- Duplicated Groups can be normally selected too
Bonus goodie: SHIFT+A (Toolbox) now has entry "Add group" too, with a
listing of all groups, allowing to add Group instances immediate.
-> Library System
- SHIFT+F4 data browse now shows the entire path for linked data
- Outliner draws Library Icons to denote linked data
- Outliner operation added: "Make Local" for library data.
- Outliner now also draws Groups in regular view, allowing to unlink too.
-> Fixes
- depsgraph missed signal update for bone-parented Objects
- on reading file, the entire database was tagged to "recalc" fully,
causing unnecessary slowdown on reading.
Might have missed stuff... :)
This first implementation only supports it for Meshes with UV maps, and
only tangents in V direction.
Tangent diffuse:
http://www.blender.org/bf/0001_0080.avi
Tangent spec, diffuse, and bump:
http://www.blender.org/bf/20001_0080.avi
NOTE: since UV coordinates are still very badly subsurfed, this won't work
well for subsurf meshes... on the todo.
On the todo;
- generate tangents for meshes without UV (with some options)
- use tangents from Curve/Surface
- add the Ashkimin shader from tracker
-----
Important bugfix; curves didn't render anymore since yesterday. :)
(As usual movies disappears after while)
Face example showing stress values on a blend. White is stretch, black
is squeeze
http://www.blender.org/bf/0001_0014.avi
Quick test with softbody stretch
http://www.blender.org/bf/0001_0100.avi
Based on the difference of the "Orco" (original undeformed coordinate)
and the actual render coordinate, a stress value is computed to make
textures react to stretching or wrinking skin.
The texture coordinate is neutral (0) on relaxed state. -1 is squeezed
to zero, +1 is stretched to infinity.
Note that scaling (object itself or parent) also will result in
stress values.
The reason for the huge commit is a cleanup in allocating memory for
the vertices. These were growing too large with new options, so now it
allocates the optional coordinates dynamically.
Saves about 20 MB memory per 1M vertices already. But best of all is that
I now can add much more fun... so tangents, here we come!
Previous experiment (in 2000) didn't satisfy, it had even some primitive
NLA option in groups... so, cleaned up the old code (removed most) and
integrated it back in a more useful way.
Usage:
- CTRL+G gives menu to add group, add to existing group, or remove from
groups.
- In Object buttons, a new (should become first) Panel was added, showing
not only Object "ID button" and Parent, but also the Groups the Object
Belongs to. These buttons also allow rename, assigning or removing.
- To indicate Objects are grouped, they're drawn in a (not theme yet, so
temporal?) green wire color.
- Use ALT+SHIFT mouse-select to (de)select an entire group
But, the real power of groups is in the following features:
-> Particle Force field and Guide control
In the "Particle Motion" Panel, you can indicate a Group name, this then
limits force fields or guides to members of that Group. (Note that layers
still work on top of that... not sure about that).
-> Light Groups
In the Material "Shaders" Panel, you can indicate a Group name to limit
lighting for the Material to lamps in this group. The Lights in a Group do
need to be 'visible' for the Scene to be rendered (as usual).
-> Group Duplicator
In the Object "Anim" Panel, you can set any Object (use Empty!) to
duplicate an entire Group. It will make copies of all Objects in that Group.
Also works for animated Objects, but it will copy the current positions or
deforms. Control over 'local timing' (so we can do Massive anims!) will be
added later.
(Note; this commit won't render Group duplicators yet, a fix in bf-blender
will enable that, next commit will sync)
-> Library Appending
In the SHIFT-F1 or SHIFT+F4 browsers, you can also find the Groups listed.
By appending or linking the Group itself, and use the Group Duplicator, you
now can animate and position linked Objects. The nice thing is that the
local saved file itself will only store the Group name that was linked, so
on a next file read, the Group Objects will be re-read as stored (changed)
in the Library file.
(Note; current implementation also "gives a base" to linked Group Objects,
to show them as Objects in the current Scene. Need that now for testing
purposes, but probably will be removed later).
-> Outliner
Outliner now shows Groups as optio too, nice to organize your data a bit too!
In General, Groups have a very good potential... for example, it could
become default for MetaBall Objects too (jiri, I can help you later on how
this works). All current 'layer relationships' in Blender should be dropped
in time, I guess...
New diffuse shader, "Fresnel", which using existing fresnel formula.
Since it hilights edges (away from lamp), nice to fill darker parts with
the new layering system.
Weird stuff though;
http://www.blender.org/bf/0001_0040.avi
(Movie disappears in a couple of days!)
Note; for ray-shadow you need to use the Bias, to prevent 'terminator'
problems. I made that option default now.
(WIP, don't bugs for this in tracker yet please!)
- New Panel "Layers" in Material buttons, allows to add unlimited amount
of materials on top of each other.
- Every Layer is actually just another Material, which gets rendered/shaded
(including texture), and then added on top of previous layer with an
operation like Mix, Add, Mult, etc.
- Layers render fully independent, so bumpmaps are not passed on to next
layers.
- Per Layer you can set if it influences Diffuse, Specular or Alpha
- If a Material returns alpha (like from texture), the alpha value is
used for adding the layers too.
- New texture "Map To" channel allows to have a texture work on a Layer
- Each layer, including basis Material, can be turned on/off individually
Notes:
- at this moment, the full shading pass happens for each layer, including
shadow, AO and raytraced mirror or transparency...
- I had to remove old hacks from preview render, which corrected reflected
normals for preview texturing.
- still needs loadsa testing!
An image texture without picture returned a '0', causing the texture code
to generate a bump normal that warped the entire shading around.
In bugreport it was showing terminator issues with ray shadow.
Terminator problems can still occor with more extreme bumpmaps though,
something to look at further.
Replaced sqrt() with the safe version sasqrt() in anisotropic.
the formula (1 - n*n) should not give negative values for n = dotproduct
of normals, but float inaccuracy can make negative zeros, as we know!
This solves, hopefully, last annoying "blotches" in hair!
raytracing with a linedist (cylinder) check. It didn't behave OK on strands
set to be more than 1 pixel (not to mention 10 pixels), and I have no time
now to fully check and code it.
Which is not really important either... In general, all 3d papers talk about
using shadowbuffer shadows for hair, which gives much nicer results for
dense volumes of polygons.
This commit restores regular (flat polygon) raytrace shadow for strands.
Also fixes some nasty bugs as reported. :)
- raytracing code now uses cylinder-line intersect for strands. It used
to intersect with the screen-aligned strand quads, giving too many
misses for tracing.
Note; mirror for hair is still not well supported!
- added in Materials a choice for whether a material is Traceable for
raytracing or whether it is part of Shadow buffers. This way you can
exclude hair strands from raytracing, but still get shadowbuffers for it.
Very ancient bug in boxfilter for image texture popped up with new code
for correct bumpmapping. Caused texture type "Extend" not to work when
using for bump.
While investigating alternative filters (Mitchell), I found two small
errors in the Gauss code, it clipped wrong and multiplied wrong, causing
settings other than filter size 1.0 to not work properly.
Took the last-minute liberty to add more filter types in Blender too.
Also wrote an extensive log about how sampling & filtering in Blender
works.
http://www.blender3d.org/cms/Samples_and_Filtering.723.0.html
Ancient issue with Image bumpmap & orientation... this report was about
UV textures, which flipped bump direction when UV coordinates rotate.
So... time to dive deeper into it, and I found a very bad old construction
in the way image textures handled bump normals. Instead of using the
correct dx,dy vectors it was calculating the boundbox for it, and applied
it as such (loosing negative direction for the dx or dy that way).
Anyhoo! To make a long story short; the weird "correct Nor Map" option
now can be removed... but requires careful testing!
Here's renders from file I got from env, 239 means old and 240 is this cvs.
http://www.blender.org/bf/dino239.jpghttp://www.blender.org/bf/dino240.jpghttp://www.blender.org/bf/dino239a.jpghttp://www.blender.org/bf/dino240a.jpg
Note the much finer detail, and the more consistant normals direction.
No, go render testing dudes!
