Shader node float precision issue cause different results on CPU and GPU #67448

New Issue

Leo Wattenberg · 2019-07-22T15:05:30+02:00

Leo Wattenberg commented

2019-07-22 15:05:30 +02:00

System Information
Operating system: Windows-10-10.0.18362 64 Bits
Graphics card: GeForce GTX 1070 with Max-Q Design/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 430.86
CPU: Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz

Blender Version
Broken: version: 2.80rc2 (2.80 (sub 74), branch: master, commit date: 2019-07-18 14:52, hash: 38d4483c6a)

Short description of error
When rendering a mandelbulb volumetric thing on cycles using CPU and GPU, the two have very different ideas on how the scene should look like, making the individual tiles clearly visible.

Render

(most tiles are GPU-rendered, note the stray red CPU-rendered tile to the left)

Blend
mandelbrot.blend

Potentially useful information

#50193 may be the same bug; I opened a new one because that bug doesn't report differences within a single frame, and is from 2016.
Since most of this object is endless math nodes, it may be some sort of floating-point thing that works differently on CPUs and GPUs.
I was following this tutorial: https://www.youtube.com/watch?v=WSQFt1Nruns
I noticed that some operations for which I expected single inputs (namely the sines) had two inputs. In the tutorial, the dude said the second input doesn't do anything, but then, that's from the alpha. Maybe taking a 0.5th sine of a value does something strange?

**System Information** Operating system: Windows-10-10.0.18362 64 Bits Graphics card: GeForce GTX 1070 with Max-Q Design/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 430.86 CPU: Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz **Blender Version** Broken: version: 2.80rc2 (2.80 (sub 74), branch: master, commit date: 2019-07-18 14:52, hash: `38d4483c6a`) **Short description of error** When rendering a mandelbulb volumetric thing on cycles using CPU and GPU, the two have very different ideas on how the scene should look like, making the individual tiles clearly visible. **Render** ![t3-rc2.png](https://archive.blender.org/developer/F7623022/t3-rc2.png) (most tiles are GPU-rendered, note the stray red CPU-rendered tile to the left) **Blend** [mandelbrot.blend](https://archive.blender.org/developer/F7623033/mandelbrot.blend) **Potentially useful information** * #50193 may be the same bug; I opened a new one because that bug doesn't report differences within a single frame, and is from 2016. * Since most of this object is endless math nodes, it may be some sort of floating-point thing that works differently on CPUs and GPUs. * I was following this tutorial: https://www.youtube.com/watch?v=WSQFt1Nruns * I noticed that some operations for which I expected single inputs (namely the sines) had two inputs. In the tutorial, the dude said the second input doesn't do anything, but then, that's from the alpha. Maybe taking a 0.5th sine of a value does something strange?

Leo Wattenberg commented

2019-07-22 15:05:30 +02:00

Added subscriber: @leoxd.mtar

blender-admin commented

2019-07-22 15:05:30 +02:00

#85988 was marked as duplicate of this issue

Michael Hermann commented

2019-07-22 15:42:57 +02:00

Added subscriber: @MichaelHermann

Michael Hermann commented

2019-07-22 15:42:57 +02:00

How much of your GPU memory is used? I'm asking because I might have a similar issue.
For me, this occurs, when GPU memory is close to full. Then, some geometry seems to be omitted from the GPU, resulting in a tiled look. I'm using Microdisplacement btw.

How much of your GPU memory is used? I'm asking because I might have a similar issue. For me, this occurs, when GPU memory is close to full. Then, some geometry seems to be omitted from the GPU, resulting in a tiled look. I'm using Microdisplacement btw.

Leo Wattenberg commented

2019-07-22 16:24:07 +02:00

GPU Memory is at 2.2GB, out of 8GB dedicated, 16GB dedicated+shared

Michael Hermann commented

2019-07-22 16:51:10 +02:00

Ok... thanks for the info. It seems unrelated.

gabe2252 commented

2019-07-22 22:07:09 +02:00

Added subscriber: @gabe2252

gabe2252 commented

2019-07-22 22:07:09 +02:00

I was going to report a similar issue I was having with volumetrics. I use a rx 580 8gb on linux (ubuntu 18.04) with the amdgpu driver (non pro) and blender 2.8 rc2

areoLux1.blend

I was going to report a similar issue I was having with volumetrics. I use a rx 580 8gb on linux (ubuntu 18.04) with the amdgpu driver (non pro) and blender 2.8 rc2 ![cpu+gpu.png](https://archive.blender.org/developer/F7623483/cpu_gpu.png) [areoLux1.blend](https://archive.blender.org/developer/F7623486/areoLux1.blend)

Ray molenkamp commented

2019-07-22 22:29:08 +02:00

Added subscriber: @LazyDodo

Ray molenkamp commented

2019-07-22 22:29:08 +02:00

try changing this setting to distance

try changing this setting to distance ![image.png](https://archive.blender.org/developer/F7623525/image.png)

Ray molenkamp commented

2019-07-22 22:37:51 +02:00

nope, that did not work either. i'll leave it for someone smarter.

Gavin Scott commented

2019-07-23 00:24:09 +02:00

Added subscriber: @GavinScott

Gavin Scott commented

2019-07-23 00:24:09 +02:00

Reproduces here (Win 10/Nvidia 1060). You can reduce the tile size and sampling and still see the issue. I note that the YouTube tutorial's final .blend file does not exhibit the problem when switched to Cycles (but then it has a somewhat different shader setup). It appears to be a very small progressive difference as it moves through each of the stages in the shader. If you remove most of them then the issue is invisible. As you add back in more of the top-level node groups, the problem kind of fades into existence and becomes more pronounced.

I looked for differences in settings, lighting etc. but nothing jumped out. I think it's probably just a very slight difference in the render kernels. It can be really hard to get identical floating-point behavior from two different code implementations.

When you use the Math node with a function like Sine that only takes a single value, the second input just stays unused and It will not matter what input it gets.

Reproduces here (Win 10/Nvidia 1060). You can reduce the tile size and sampling and still see the issue. I note that the YouTube tutorial's final .blend file does not exhibit the problem when switched to Cycles (but then it has a somewhat different shader setup). It appears to be a very small progressive difference as it moves through each of the stages in the shader. If you remove most of them then the issue is invisible. As you add back in more of the top-level node groups, the problem kind of fades into existence and becomes more pronounced. I looked for differences in settings, lighting etc. but nothing jumped out. I think it's probably just a very slight difference in the render kernels. It can be *really* hard to get identical floating-point behavior from two different code implementations. When you use the Math node with a function like Sine that only takes a single value, the second input just stays unused and It will not matter what input it gets.

Sergey Sharybin commented

2019-07-25 15:38:43 +02:00

Added subscribers: @brecht, @Sergey

Brecht Van Lommel was assigned by Sergey Sharybin

2019-07-25 15:38:43 +02:00

Sergey Sharybin commented

2019-07-25 15:38:43 +02:00

There is actually a code to ensure Distance sampling is used for all shaders when there are both CPU and GPU are involved, which seems to work the way it is designed to.

I can not reproduce the issue with the mandelbrot file. And it doesn't look like something what could be caused by different shading capabilities on GPU and CPU. But i can not reproduce the issue.

The aeroLux1 issue is that there are two volume domains, which are exactly matching together. It is possible that due to different precision on CPU and GPU one of the intersections is missing when ray exits the domains, causing integration loop to discard volume to avoid overblown pixel due to missive texture.
Just make one of the domains slightly bigger.
Not sure we can do anything about it.

@brecht, mind trying to reproduce the first issue?

There is actually a code to ensure Distance sampling is used for all shaders when there are both CPU and GPU are involved, which seems to work the way it is designed to. I can not reproduce the issue with the mandelbrot file. And it doesn't look like something what could be caused by different shading capabilities on GPU and CPU. But i can not reproduce the issue. The aeroLux1 issue is that there are two volume domains, which are exactly matching together. It is possible that due to different precision on CPU and GPU one of the intersections is missing when ray exits the domains, causing integration loop to discard volume to avoid overblown pixel due to missive texture. Just make one of the domains slightly bigger. Not sure we can do anything about it. @brecht, mind trying to reproduce the first issue?

Michael Hermann commented

2019-07-28 12:00:53 +02:00

Hm... now I'm not sure. I'll just post these pictures and let Devs decide if this could be related.
My scene has Volumes in there, but I disabled them to pinpoint the problem. DoF or Volumes don't seem to make a difference for this.

This first image shows the issue. The GPU tiles omit some geometry in front. (See left side of the jaw for example.)

In this second image I just pulled the camera back a little. Since my scene uses Microdisplacement, this reduced the memory consumption just below the critical point and the image looks fine. (I just stopped the render because it takes quite a long time)

I'm rendering with a 1080Ti with 11GB of memory. The second image needed about 9.3GB. Just above that, it starts looking like the first image.

If this is helpful to the issue or I can provide anything else, please let me know.
I can provide the .blend file, but relatively heavy and probably depending on the hardware if the issue shows up or not. Let me know if you believe it's helpful or if I can provide anything else.

Hm... now I'm not sure. I'll just post these pictures and let Devs decide if this could be related. My scene has Volumes in there, but I disabled them to pinpoint the problem. DoF or Volumes don't seem to make a difference for this. This first image shows the issue. The GPU tiles omit some geometry in front. (See left side of the jaw for example.) ![bug4_noDOFnoVolume.png](https://archive.blender.org/developer/F7633055/bug4_noDOFnoVolume.png) In this second image I just pulled the camera back a little. Since my scene uses Microdisplacement, this reduced the memory consumption just below the critical point and the image looks fine. (I just stopped the render because it takes quite a long time) ![bug5_noDOFnoVolume_9.3GBMem_CamMovedBack.png](https://archive.blender.org/developer/F7633054/bug5_noDOFnoVolume_9.3GBMem_CamMovedBack.png) I'm rendering with a 1080Ti with 11GB of memory. The second image needed about 9.3GB. Just above that, it starts looking like the first image. If this is helpful to the issue or I can provide anything else, please let me know. I can provide the .blend file, but relatively heavy and probably depending on the hardware if the issue shows up or not. Let me know if you believe it's helpful or if I can provide anything else.

Cole commented

2019-08-03 21:42:37 +02:00

Added subscriber: @ColeMorris

Brecht Van Lommel commented

2019-09-25 17:32:31 +02:00

I can reproduce the mandelbrot issue. The other issues reported here seem unrelated, and should be reported separately.

This is a small precision difference between math on the CPU and GPU in one of the shader nodes, which gets worse with every iteration until it has a big visible effect.

We need to figure out which node and operation exactly causes the issue. But we may not be able to fix this, the math function implementations on the GPU may be slightly different and we are not likely to implement our own, because of the performance impact.

I can reproduce the mandelbrot issue. The other issues reported here seem unrelated, and should be reported separately. This is a small precision difference between math on the CPU and GPU in one of the shader nodes, which gets worse with every iteration until it has a big visible effect. We need to figure out which node and operation exactly causes the issue. But we may not be able to fix this, the math function implementations on the GPU may be slightly different and we are not likely to implement our own, because of the performance impact.

Brecht Van Lommel was unassigned by Dalai Felinto

2019-12-23 16:33:37 +01:00

Brecht Van Lommel changed title from ~~GPU-rendered tiles look different than CPU-rendered ones~~ to Shader node float precision issue cause different results on CPU and GPU

2020-01-27 11:36:35 +01:00

Connor M. Denning commented

2022-03-22 04:43:12 +01:00

Added subscriber: @Connor-Denning

Germano Cavalcante commented

2022-07-07 16:06:56 +02:00

Added subscribers: @olsarxd, @mano-wii, @lichtwerk

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Shader node float precision issue cause different results on CPU and GPU #67448