After testing and feedback, I've decided to slightly modify the way color
management works internally. While the previous method worked well for
rendering, was a smaller transition and had some advantages over this
new method, it was a bit more ambiguous, and was making things difficult
for other areas such as compositing.
This implementation now considers all color data (with only a couple of
exceptions such as brush colors) to be stored in linear RGB color space,
rather than sRGB as previously. This brings it in line with Nuke, which also
operates this way, quite successfully. Color swatches, pickers, color ramp
display are now gamma corrected to display gamma so you can see what
you're doing, but the numbers themselves are considered linear. This
makes understanding blending modes more clear (a 0.5 value on overlay
will not change the result now) as well as making color swatches act more
predictably in the compositor, however bringing over color values from
applications like photoshop or gimp, that operate in a gamma space,
will give identical results.
This commit will convert over existing files saved by earlier 2.5 versions to
work generally the same, though there may be some slight differences with
things like textures. Now that we're set on changing other areas of shading,
this won't be too disruptive overall.
I've made a diagram explaining the pipeline here:
http://mke3.net/blender/devel/2.5/25_linear_workflow_pipeline.png
and some docs here:
http://www.blender.org/development/release-logs/blender-250/color-management/
* Convert all code to use new functions.
* Branch maintainers may want to skip this commit, and run this
conversion script instead, if they use a lot of math functions
in new code:
http://www.pasteall.org/9052/python
- 1st stage: Linear Workflow
This implements automatic linear workflow in Blender's renderer. With the
new Colour Management option on in the Render buttons, all inputs to the
renderer and compositor are converted to linear colour space before
rendering, and gamma corrected afterwards. In essence, this makes all
manual gamma correction with nodes, etc unnecessary, since it's done
automatically through the pipeline.
It's all explained much better in the notes/doc here, so please have a look:
http://wiki.blender.org/index.php/Dev:Source/Blender/Architecture/Colour_Management
And an example of the sort of difference it makes:
http://mke3.net/blender/devel/rendering/b25_colormanagement_test01.jpg
This also enables Colour Management in the default B.blend, and changes the
default lamp falloff to inverse square, which is more correct, and much
easier to use now it's all gamma corrected properly.
Next step is to look into profiles/soft proofing for the compositor.
Thanks to brecht for reviewing and fixing some oversights!
SunSky didn't include skycolor in raytrace.
Note: there seems to be an error in sunsky when looking straight down,
so this option requires raytracing stuff not in outer space. :)
- removed ugly clamping function (it was dividing XYZ based on max of
one of the values)
- added option to use Exposure, this only works for brightness (Y).
results look very pleasant, foggy and hazy results are possible.
with exposre==0, no exposure happens for HDR extreme range skies,
this is how yafray rendered it.
- added menu for choosing color spaces (CIE = modern lcds)
Please review! (and yes i know it's still not in World :)
- Added blending mode and factor option, so it's more clear and
controllable what happens with it. Also nice for crazy effects
of course!
- Preview render now shows preview for it too
On the todos:
- have this in World buttons (as well) for quicker sky setups
- review math of color clamping and scaling, this is definitely
not good... but a fix will make old files look very different.
Finally, after a long time new render candy for the non-game peoples! :)
Good doc is here: (url splits in two)
http://www.harkyman.com/2008/08/06/controllable-shadow-intensity-
and-color/
Note the colorpicker for shadow is in "Shadow and Spot" panel. A bit
hidden, could get more attention. For later. :)
This adds a few settings to control global render quality, for faster
renders when tweaking lighting etc. The implementation is not so great,
and this should really be part of a proper render profile and preset
system. So for now it's a hidden Peach feature, enabled by setting rt
to 1. Before the next release, I'll either remove or improve it.
Settings are:
- Maximum subsurf level
- Child particles percentage
- Maximum shadow map samples
- AO and SSS quality factor
Lampbuffers require painful bias tweaking (to prevent aliasing or to
get shadow detail). Sometimes you want this different per object, like
for gras you want less shadow detail, but for the ground you want high
detail. This feature allows to tweak it.
The new "LBias" slider is in shader panel, bottom. Ugly! But, thats for
later...
[#6861] Black dots when using small lamps on Mirror materials in 2.44 and 2.43.
some values were not initialized properly, for example, the window coordinates for reflections, this caused NAN color values for some pixels, (may also fix plumiferos bad pixel problem from last bconf)
- AO "Use Distances" can not use 'sky texture' for colors, that bug was
fixed in UI (prevent settings to combine), but better also gets fixed
on reading files.
- Backbuf render, when not existed, didn't initialize backdrop color to 0.
Thread render error: a flag was stored in read-only data to indicate
whether top or bottom hemisphere of sky was rendered. That can't work
with tiles (but did work when scanlines were threads).
The Scene audio.mixrate variable was zero on creating new scenes, causing
synced playback to fail.
Added initialize in kernel, and patch in buttons drawing to set the value
for wrongly saved files.
(Pixelshading: small correction for more optimal code)
"Backbuf" image render option is back :)
Nicer coded, using Image texture functions.
If you want 100% reliable pixel to pixel accuracy you have to use
compositing. (Old Backbuf didn't do this accuracy either btw).
The old implementation was added quite hackish (talking about 10 yr ago).
You also had to make a small image slice, which was extended Xparts in
size. That also required to adjust the camera angle. Very clumsy.
Now; when enabling the Panorama option, it will automatically apply the
panorama effect on the vertically aligned tiles. You can just enable or
disable the "Pano" button, to get a subtle lens effect like this:
(without pano)
http://www.blender.org/bf/rt.jpg
(with pano)
http://www.blender.org/bf/rt1.jpg
For Panorama render, the minimum slice size has been hardcoded to be 8
pixels. The XParts button goes up to 512 to allow that. In practice,
rendering 64 slices will already give very good images for a wide angle
lens of 90 degrees, the curvature of straight lines then is equal to
a circle of 256 points.
Rendering a full 360 degree panorama you do by creating an extreme wide
angle camera. The theory says camera-lens 5 should do 360 degrees, but
for some reason my tests reveil it's 5.1... there's a rounding error
somewhere, maybe related to the clipping plane start? Will look at that
later. :)
Also note that for each Xpart slice, the entire database needs to be
rotated around camera to correct for panorama, on huge scenes that might
give some overhead.
Threaded render goes fine for Panorama too, but it can only render the
vertically aligned parts in parallel. For the next panorama slice it has
to wait for all threads of the current slice to be ready.
On reading old files, I convert the settings to match as closely as
possible the new situation.
Since I cannot bump up the version #, the code detects for old panorama
by checking for the image size. If image width is smaller than height, it
assumes it's an old file (only if Panoroma option was set).
- Composit cache now gets fully freed on a render. Each output socket of a
node stores the entire image... and while render that's a waste of memory
- Sky 'paper' render was using wrong texture coordinates
- Found missing test_break() in ztransp rendering.
A full detailed description of this will be done later... is several days
of work. Here's a summary:
Render:
- Full cleanup of render code, removing *all* globals and bad level calls
all over blender. Render module is now not called abusive anymore
- API-fied calls to rendering
- Full recode of internal render pipeline. Is now rendering tiles by
default, prepared for much smarter 'bucket' render later.
- Each thread now can render a full part
- Renders were tested with 4 threads, goes fine, apart from some lookup
tables in softshadow and AO still
- Rendering is prepared to do multiple layers and passes
- No single 32 bits trick in render code anymore, all 100% floats now.
Writing images/movies
- moved writing images to blender kernel (bye bye 'schrijfplaatje'!)
- made a new Movie handle system, also in kernel. This will enable much
easier use of movies in Blender
PreviewRender:
- Using new render API, previewrender (in buttons) now uses regular render
code to generate images.
- new datafile 'preview.blend.c' has the preview scenes in it
- previews get rendered in exact displayed size (1 pixel = 1 pixel)
3D Preview render
- new; press Pkey in 3d window, for a panel that continuously renders
(pkey is for games, i know... but we dont do that in orange now!)
- this render works nearly identical to buttons-preview render, so it stops
rendering on any event (mouse, keyboard, etc)
- on moving/scaling the panel, the render code doesn't recreate all geometry
- same for shifting/panning view
- all other operations (now) regenerate the full render database still.
- this is WIP... but big fun, especially for simple scenes!
Compositor
- Using same node system as now in use for shaders, you can composit images
- works pretty straightforward... needs much more options/tools and integration
with rendering still
- is not threaded yet, nor is so smart to only recalculate changes... will be
done soon!
- the "Render Result" node will get all layers/passes as output sockets
- The "Output" node renders to a builtin image, which you can view in the Image
window. (yes, output nodes to render-result, and to files, is on the list!)
The Bad News
- "Unified Render" is removed. It might come back in some stage, but this
system should be built from scratch. I can't really understand this code...
I expect it is not much needed, especially with advanced layer/passes
control
- Panorama render, Field render, Motion blur, is not coded yet... (I had to
recode every single feature in render, so...!)
- Lens Flare is also not back... needs total revision, might become composit
effect though (using zbuffer for visibility)
- Part render is gone! (well, thats obvious, its default now).
- The render window is only restored with limited functionality... I am going
to check first the option to render to a Image window, so Blender can become
a true single-window application. :)
For example, the 'Spare render buffer' (jkey) doesnt work.
- Render with border, now default creates a smaller image
- No zbuffers are written yet... on the todo!
- Scons files and MSVC will need work to get compiling again
OK... thats what I can quickly recall. Now go compiling!
- Image textures use float colors now, when present. Works for mipmap too,
and for AO skycolor (probes)
- Backbuffer option uses float buffers too. Note that rendering OSA will
resample the backbuffer, filtering it... will need to be solved with the
new composit stage
- LMB sampling in image window now shows float color too
+ bugfix in imbuf, filtering for float buffers had an error.
Previous experiment (in 2000) didn't satisfy, it had even some primitive
NLA option in groups... so, cleaned up the old code (removed most) and
integrated it back in a more useful way.
Usage:
- CTRL+G gives menu to add group, add to existing group, or remove from
groups.
- In Object buttons, a new (should become first) Panel was added, showing
not only Object "ID button" and Parent, but also the Groups the Object
Belongs to. These buttons also allow rename, assigning or removing.
- To indicate Objects are grouped, they're drawn in a (not theme yet, so
temporal?) green wire color.
- Use ALT+SHIFT mouse-select to (de)select an entire group
But, the real power of groups is in the following features:
-> Particle Force field and Guide control
In the "Particle Motion" Panel, you can indicate a Group name, this then
limits force fields or guides to members of that Group. (Note that layers
still work on top of that... not sure about that).
-> Light Groups
In the Material "Shaders" Panel, you can indicate a Group name to limit
lighting for the Material to lamps in this group. The Lights in a Group do
need to be 'visible' for the Scene to be rendered (as usual).
-> Group Duplicator
In the Object "Anim" Panel, you can set any Object (use Empty!) to
duplicate an entire Group. It will make copies of all Objects in that Group.
Also works for animated Objects, but it will copy the current positions or
deforms. Control over 'local timing' (so we can do Massive anims!) will be
added later.
(Note; this commit won't render Group duplicators yet, a fix in bf-blender
will enable that, next commit will sync)
-> Library Appending
In the SHIFT-F1 or SHIFT+F4 browsers, you can also find the Groups listed.
By appending or linking the Group itself, and use the Group Duplicator, you
now can animate and position linked Objects. The nice thing is that the
local saved file itself will only store the Group name that was linked, so
on a next file read, the Group Objects will be re-read as stored (changed)
in the Library file.
(Note; current implementation also "gives a base" to linked Group Objects,
to show them as Objects in the current Scene. Need that now for testing
purposes, but probably will be removed later).
-> Outliner
Outliner now shows Groups as optio too, nice to organize your data a bit too!
In General, Groups have a very good potential... for example, it could
become default for MetaBall Objects too (jiri, I can help you later on how
this works). All current 'layer relationships' in Blender should be dropped
in time, I guess...
This fixes the active object in place when orbiting the view.
Choppy 15fps demo can be seen there: http://www.elysiun.com/~theeth/bf/around_active.html
Image Memory Grabage Collection
This adds memory handling to the image code. An image is tagged each time it is used.
During a collection cycle (frequency of cycles is user defined), if an image is older
than a user defined limit, its buffer gets deallocated. This also applies to gl memory buffers.
Images that are loading in GL memory needs to go through two time outs before being fully deallocated: the first time out deallocated the gl memorry, the second the buffer in ram.
Notes:
Image buffer loaded from python gets tagged as permanent upon load. That tag is removed when python stops using the image.
I might have missed some tagging spots, especially in the rendering pipeline. Someone with more knowledge about this code should check to be careful.
Tagging is done on every access, for rendering, this will probably be a performance hit. A scheme should be developped to only tag when the rendering is completed.
Collecting is called in draw_object, most likely not the best place to do it.
Safe from undo, since using undo deallocates memory anyway (like when loading a blend file with one currently opened)
Userpref DNA changes:
I've changed a couple of flagging variables from short to int. Some because they needed more space, others to keep SDNA happy.
Info window changes:
I've grouped a couple of buttons in aligned blocks and changed the color of mutually exclusive options to make them clearer.
Matt didn't do any changes on that in tuhopuu, so hopefully I'm not stepping on anyone's feet with this.
Also changed double constants into floats with f in a couple of places (mostly space.c) to make compiler happier.
alone with the following flags :
-Wall -Wno-char-subscripts -Wno-missing-braces.
the only one still worrying me is in rand.c line 57 :
rand.c:57: integer constant is too large for "long" type
but i have no clue about how correct cross-compiler and 32/64 bits friendly
see also my mail to commiter list for signed/unsigned issues
- removed ugly pointerhack from OSA render (negative indices denoted
pointers). this should solve memory errors when using >1.5 gig mem
- cleaned up usage of zbuffer values. These are signed in Blender, and
treated as unsigned all over, giving confusing code
- fixed incorrect gamma-adding for halos (caused in after xmas commit)
And bugfix #2101; wire render didn't give correct rendering for mist.
This caused by fact wires are 2D pixel lines, and not correctly filled
in faces. Retrieving the 3d coordinate while render cannot use a face-
equation then. Solved by retrieving 3D coordinate based on zbuffer value.
Still todo here: calculating correct texture coordinates for wire-edges
that are no faces.
Render:
- New; support for dual CPU render (SDL thread)
Currently only works with alternating scanlines, but gives excellent
performance. For both normal render as unified implemented.
Note the "mutex" locks on z-transp buffer render and imbuf loads.
- This has been made possible by major cleanups in render code, especially
getting rid of globals (example Tin Tr Tg Tb Ta for textures) or struct
OSA or using Materials or Texture data to write to.
- Made normal render fully 4x32 floats too, and removed all old optimizes
with chars or shorts.
- Made normal render and unified render use same code for sky and halo
render, giving equal (and better) results for halo render. Old render
now also uses PostProcess options (brightness, mul, gamma)
- Added option ("FBuf") in F10 Output Panel, this keeps a 4x32 bits buffer
after render. Using PostProcess menu you will note an immediate re-
display of image too (32 bits RGBA)
- Added "Hue" and "Saturation" sliders to PostProcess options
- Render module is still not having a "nice" API, but amount of dependencies
went down a lot. Next todo: remove abusive "previewrender" code.
The last main global in Render (struct Render) now can be re-used for fully
controlling a render, to allow multiple "instances" of render to open.
- Renderwindow now displays a smal bar on top with the stats, and keeps the
stats after render too. Including "spare" page support.
Not only easier visible that way, but also to remove the awkward code that
was drawing stats in the Info header (extreme slow on some ATIs too)
- Cleaned up blendef.h and BKE_utildefines.h, these two had overlapping
defines.
- I might have forgotten stuff... and will write a nice doc on the architecture!
Unified render added sky with alpha=1. Found the way how to fix it, didnt
dare that yesterday. :)
Now sky is rendered with alpha=0, like normal render.
Thanks to the recent reports, a lot of old issues with unified render have
been solved. This is going to be pretty OK!
Halos with Flare option didn't render correct in Unified render, they were
added twice!
Also noticed errors in applying alpha for 'Premul' alpha in Unified. Fixed.
For composting sky behind alpha, a gamma corrected alpha_under was used,
but Blender didnt make the gamma tables unless OSA was set.
Old bug!
Changed; while rendering with "Backbuf" that doesn't exist, blender still
renders, but without backfbuf now. It used to stop render and return, with
only a print in console... confusing.
different objects shouldn't share flags this way (still sharing of
other mesh flags in renderer... ickity pickity, but I'm not fixing now)
- removed some unnecessary uses of DNA_mesh_types.h
Main target was to make the inner rendering loop using no globals anymore.
This is essential for proper usage while raytracing, it caused a lot of
hacks in the raycode as well, which even didn't work correctly for all
situations (textures especially).
Done this by creating a new local struct RenderInput, which replaces usage
of the global struct Render R. The latter now only is used to denote
image size, viewmatrix, and the like.
Making the inner render loops using no globals caused 1000s of vars to
be changed... but the result definitely is much nicer code, which enables
making 'real' shaders in a next stage.
It also enabled me to remove the hacks from ray.c
Then i went to the task of removing redundant code. Especially the calculus
of texture coords took place (identical) in three locations.
Most obvious is the change in the unified render part, which is much less
code now; it uses the same rendering routines as normal render now.
(Note; not for halos yet!)
I also removed 6 files called 'shadowbuffer' something. This was experimen-
tal stuff from NaN days. And again saved a lot of double used code.
Finally I went over the blenkernel and blender/src calls to render stuff.
Here the same local data is used now, resulting in less dependency.
I also moved render-texture to the render module, this was still in Kernel.
(new file: texture.c)
So! After this commit I will check on the autofiles, to try to fix that.
MSVC people have to do it themselves.
This commit will need quite some testing help, but I'm around!
- fixed bug in halo rendering combined with spothalo
- fixed bug in no antialiasing with sky with spothalo
- cleaned some weird usage of gamma for spothalo and sky
(when rendering sky + spothalo, sky got gamma corrected
- removed old test code and #ifdefs for code readability
On test scenes here unified render looks much better. still some minor
issues with antialiasing... cant pin that down yet.
