Cycles: Cuda 9.2 and 10 Issues. #56858

New Issue

Ray molenkamp · 2018-09-20T21:49:37+02:00

Ray molenkamp commented

2018-09-20 21:49:37 +02:00

Cycles: Cuda 9.2 and 10 Issues.

Cuda92 has some issues with cycles, that shown most when rendering bmw2.7 gpu

I am sure we all hoped, that cuda 10 would fix this, which it didn't.

I played around with the usual suspects, aka don't inline this and even tried
disabling all inlining all together. didn't help.

then I played with the optimizers (both nvcc and ptax have their own optimizers
which you can independently control) and the issue seems to occur whenever you
call ptxas with O1 or higher (O0 makes a working, kernel) however there's no
documentation what optimizations O1 enables compared to O0 or anyway to enable
specific optimizer stages. so that turned out a dead end as well.

Then I figured if we know were the problem is we might be able to find a work
around. (my idea was to disable the optimizer for that one function, which you can't
for some reason , still it be nice to know where the issue is)

New problem: bmw27 has a lot of parts to it so after taking out various bits I ended
up with these 2 cases.

error.blend: matches the noise pattern you see in bmw27 it's a simple scene but there's
still a couple of mix shaders to it, simpler would be nicer..

error2.blend: no noise, but it renders completely black on the gpu, no fancy shader graphs
it's a straight up glass shader..

So I'm pretty sure it's something in the glass bsdf, but i don't know the code well enough
to look further into this.

error2.blend
error.blend

Cycles: Cuda 9.2 and 10 Issues. Cuda92 has some issues with cycles, that shown most when rendering bmw2.7 gpu I am sure we all hoped, that cuda 10 would fix this, which it didn't. I played around with the usual suspects, aka don't inline this and even tried disabling all inlining all together. didn't help. then I played with the optimizers (both nvcc and ptax have their own optimizers which you can independently control) and the issue seems to occur whenever you call ptxas with O1 or higher (O0 makes a working, kernel) however there's no documentation what optimizations O1 enables compared to O0 or anyway to enable specific optimizer stages. so that turned out a dead end as well. Then I figured if we know were the problem is we might be able to find a work around. (my idea was to disable the optimizer for that one function, which you can't for some reason , still it be nice to know where the issue is) New problem: bmw27 has a lot of parts to it so after taking out various bits I ended up with these 2 cases. error.blend: matches the noise pattern you see in bmw27 it's a simple scene but there's still a couple of mix shaders to it, simpler would be nicer.. ![error1_viewport.png](https://archive.blender.org/developer/F4768547/error1_viewport.png) ![error1_f12.png](https://archive.blender.org/developer/F4768552/error1_f12.png) error2.blend: no noise, but it renders completely black on the gpu, no fancy shader graphs it's a straight up glass shader.. ![error2_viewport.png](https://archive.blender.org/developer/F4768557/error2_viewport.png) ![error2_viewport_cpu.png](https://archive.blender.org/developer/F4768561/error2_viewport_cpu.png) So I'm pretty sure it's something in the glass bsdf, but i don't know the code well enough to look further into this. [error2.blend](https://archive.blender.org/developer/F4768543/error2.blend) [error.blend](https://archive.blender.org/developer/F4768544/error.blend)

Ray molenkamp commented

2018-09-20 21:49:37 +02:00

Added subscriber: @LazyDodo

Steffen Dünner commented

2018-09-20 22:21:38 +02:00

Added subscriber: @SteffenD

Brecht Van Lommel was assigned by Bastien Montagne

2018-09-21 09:46:45 +02:00

unblockedgamess commented

2018-09-22 19:58:27 +02:00

(Migrated from localhost:3001)

Added subscriber: @unblockedgamess

unblockedgamess commented

2018-09-22 19:58:27 +02:00

(Migrated from localhost:3001)

This comment was removed by @unblockedgamess

*This comment was removed by @unblockedgamess*

Seth Thompson commented

2018-09-23 13:41:10 +02:00

Added subscriber: @s3th.thompson

Wojciech Matyjewicz commented

2018-11-22 01:12:51 +01:00

Added subscriber: @wmatyjewicz

Brecht Van Lommel commented

2019-01-17 01:20:54 +01:00

I've been told the next minor CUDA 10 release will have a bugfix for this. It's a code generation bug and depends on the optimization level and function inlining. Hopefully it solves the problem on for all GPU architectures we support, we'll need to test that in detail when it comes out.

Brecht Van Lommel commented

2019-02-12 15:46:37 +01:00

I have some test builds to verify if this bug is indeed solved with the upcoming CUDA bugfix.

It would help if users can test if these render the BMW scene correctly. If it's wrong there will be very clear artifacts. There are nvcc and nvrtc builds, both need to be tested.
https://download.blender.org/ftp/brecht/cuda_bug_T56858/
https://download.blender.org/demo/test/BMW27_2.blend.zip

Particularly GTX 6xx and 7xx would be useful, since I don't have a card from that generation.

I have some test builds to verify if this bug is indeed solved with the upcoming CUDA bugfix. It would help if users can test if these render the BMW scene correctly. If it's wrong there will be very clear artifacts. There are nvcc and nvrtc builds, both need to be tested. https://download.blender.org/ftp/brecht/cuda_bug_T56858/ https://download.blender.org/demo/test/BMW27_2.blend.zip Particularly GTX 6xx and 7xx would be useful, since I don't have a card from that generation.

Ray molenkamp commented

2019-02-12 15:50:46 +01:00

is the bmw scene linked different from the one in our benchmark suite? or was it just linked for convenience?

Brecht Van Lommel commented

2019-02-12 15:51:40 +01:00

Just for convenience, should make no difference which one is tested.

Ray molenkamp commented

2019-02-12 16:09:38 +01:00

on win7 with sm_30 (gtx670) which was one of the problematic ones

nvrtc RenderTime:04:25.32

nvcc: RenderTime:04:31.64

Also the 2 sample files above render ok on both.

Soo yeah seems solved.

on win7 with sm_30 (gtx670) which was one of the problematic ones nvrtc RenderTime:04:25.32 ![bmw27_gpu0001_nvrtc.jpg](https://archive.blender.org/developer/F6590768/bmw27_gpu0001_nvrtc.jpg) nvcc: RenderTime:04:31.64 ![bmw27_gpu0001_nvcc.jpg](https://archive.blender.org/developer/F6590769/bmw27_gpu0001_nvcc.jpg) Also the 2 sample files above render ok on both. Soo yeah seems solved.

Brecht Van Lommel commented

2019-02-12 16:54:10 +01:00

I confirmed it works well on an RTX 5000 and GTX 1080, so it indeed seems to be fully resolved. Performance also seems a few % better than 10.0 for me.

So now we wait until the next CUDA release is officially out, and then upgrade all platforms to it so we don't need to use both 9.1 and 10.0 toolkits anymore.

I confirmed it works well on an RTX 5000 and GTX 1080, so it indeed seems to be fully resolved. Performance also seems a few % better than 10.0 for me. So now we wait until the next CUDA release is officially out, and then upgrade all platforms to it so we don't need to use both 9.1 and 10.0 toolkits anymore.

Ray molenkamp commented

2019-02-27 21:15:28 +01:00

10.1 is out!