This was caused by 2 things: Shadow map bias and aliasing.
It made the expected depth of the shadowmap further than the surface
itself in some cases. In normal time this leads to light leaking on normal
shadow mapping but here we need to always have the shadowmap depth above
the shading point.
To fix this, we use a 5 tap inflate filter using the minimum depth of all
5 samples. Using these 5 taps, we can deduce entrance surface derivatives
and there orientation towards the light ray. We use these derivatives to
bias the depth to avoid wrong depth at depth discontinuity in the shadowmap.
This bias can lead to some shadowleaks that are less distracting than the
lightleaks it fixes.
We also add a small bias to counteract the shadowmap depth precision.
