Cycles CPU rendering performance regression 2.91 and later #83011

New Issue

slwk1d · 2020-11-25T18:21:05+01:00

slwk1d commented

2020-11-25 18:21:05 +01:00

System Information
Operating system: Windows-10-10.0.19041-SP0 64 Bits
Graphics card: GeForce GTX 1080 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 457.30

Blender Version
Broken: version:

2.91.0, branch: master, commit date: 2020-11-25 08:34, hash: 0f45cab862
2.92.0, branch: master, commit date: 2020-11-24 05:21, hash: 037ce662e5
Worked: 2.90.1

Short description of error
I encountered some major rendering time downgrade from 2.90.1 to 2.91, resulting in the render time being doubled. With this scene included, the render time goes from 01:34 min (2.90) to 02:36 min (2.91).
With Optix denoiser enabled, the scene goes from02:11 min (2.90) to 04:30 min!! (2.91).

Exact steps for others to reproduce the error
With the file included, render the scene with 2.90 and then render the same scene with 2.91
render bug.blend

**System Information** Operating system: Windows-10-10.0.19041-SP0 64 Bits Graphics card: GeForce GTX 1080 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 457.30 **Blender Version** Broken: version: - 2.91.0, branch: master, commit date: 2020-11-25 08:34, hash: `0f45cab862` - 2.92.0, branch: master, commit date: 2020-11-24 05:21, hash: `037ce662e5` Worked: 2.90.1 **Short description of error** I encountered some major rendering time downgrade from 2.90.1 to 2.91, resulting in the render time being doubled. With this scene included, the render time goes from **01:34 min** (2.90) to **02:36 min** (2.91). With Optix denoiser enabled, the scene goes from**02:11 min** (2.90) to **04:30 min!!** (2.91). **Exact steps for others to reproduce the error** With the file included, render the scene with 2.90 and then render the same scene with 2.91 [render bug.blend](https://archive.blender.org/developer/F9373770/render_bug.blend)

slwk1d commented

2020-11-25 18:21:05 +01:00

Added subscriber: @Slowwkidd

Robert Guetzkow commented

2020-11-25 20:42:20 +01:00

Added subscriber: @rjg

Robert Guetzkow commented

2020-11-25 20:42:20 +01:00

I'm checking this. The performance of CPU rendering seems to be slower which also impacts CPU+GPU rendering.

Robert Guetzkow commented

2020-11-25 20:53:49 +01:00

The performance of CPU+GPU rendering appears to be slower in 2.91 and more sensitive to the chosen tile size. The slowdown seems to come from the tiles rendered and denoised by the CPU. It seems significant enough to warrant further investigation.

Please keep in mind that the very first time rendering with CUDA in an new version takes longer because it has to prepare the render kernels. The render times do fluctuate quite a bit between different runs in the same version, so a proper comparison would have to profile several renders.

The following quick measurements have been made using a GeForce GTX 1080 Ti and i7-6850K.

Tile size: 63x64

	CPU	CPU + OptiX denoise	GPU	GPU + OptiX denoise	CPU + GPU	CPU + GPU + OptiX denoise
	----	----	-----	-----------------------	-------------	--------------------------------
2.90.1	03:49:49 min	#81116	01:15:21 min	01:48:78 min	01:18:10 min	01:45:87 min
2.91.0	08:37:62 min	09:18:00 min	01:15:92 min	01:48:12 min	02:09:81 min	02:11:33 min
2.92.0	08:21:72 min	09:15:53 min	01:12:78 min	01:46:12 min	01:23:48 min	01:44:60 min

Tile size: 256x256

	CPU	CPU + OptiX denoise	GPU	GPU + OptiX denoise	CPU + GPU	CPU + GPU + OptiX denoise
	----	----	-----	-----------------------	-------------	--------------------------------
2.90.1	04:13:60 min	#81116	01:21:51 min	01:26:24 min	03:01:63 min	02:59:08 min
2.91.0	09:13:60 min	09:41:70 min	01:21:40 min	01:25:98 min	15:32:59 min	15:57:15 min
2.92.0	10:54:25 min	11:05:27 min	01:33:67 min	01:38:92 min	01:54:19 min	02:21:31 min

NOTE: Blender 2.92 includes the tile stealing feature , which masks the performance regression when using CPU+GPU rendering.

The performance of CPU+GPU rendering appears to be slower in 2.91 and more sensitive to the chosen tile size. The slowdown seems to come from the tiles rendered and denoised by the CPU. It seems significant enough to warrant further investigation. Please keep in mind that the very first time rendering with CUDA in an new version takes longer because it has to prepare the render kernels. The render times do fluctuate quite a bit between different runs in the same version, so a proper comparison would have to profile several renders. The following quick measurements have been made using a GeForce GTX 1080 Ti and i7-6850K. **Tile size: 63x64** | | CPU | CPU + OptiX denoise | GPU | GPU + OptiX denoise | CPU + GPU | CPU + GPU + OptiX denoise | | -- | -- | -- | -- | -- | -- | -- | | | ---- | ---- | ----- | ----------------------- | ------------- | -------------------------------- | | 2.90.1 | **03:49:49 min** | #81116 | 01:15:21 min | 01:48:78 min | 01:18:10 min | 01:45:87 min| | 2.91.0 | **08:37:62 min** | 09:18:00 min | 01:15:92 min | 01:48:12 min | **02:09:81 min** | **02:11:33 min** | | 2.92.0 | **08:21:72 min** | 09:15:53 min | 01:12:78 min | 01:46:12 min | 01:23:48 min | 01:44:60 min | **Tile size: 256x256** | | CPU | CPU + OptiX denoise | GPU | GPU + OptiX denoise | CPU + GPU | CPU + GPU + OptiX denoise | | -- | -- | -- | -- | -- | -- | -- | | | ---- | ---- | ----- | ----------------------- | ------------- | -------------------------------- | | 2.90.1 | **04:13:60 min** | #81116 | 01:21:51 min | 01:26:24 min | 03:01:63 min | 02:59:08 min | | 2.91.0 | **09:13:60 min** | 09:41:70 min | 01:21:40 min | 01:25:98 min | **15:32:59 min** | **15:57:15 min** | | 2.92.0 | **10:54:25 min** | 11:05:27 min | 01:33:67 min | 01:38:92 min | 01:54:19 min | 02:21:31 min | NOTE: Blender 2.92 includes the [tile stealing feature ](https://developer.blender.org/rB517ff40b124bc9d1324ccf7561a59ac51bf86602), which masks the performance regression when using CPU+GPU rendering.

slwk1d commented

2020-11-25 21:05:58 +01:00

@rjg Thanks for looking into it, hope it will eventually be resolved!
The time measurements I presented weren't the first render after starting the program, but the second/third, anyway I'll do some other tests just to be sure and write them down here.
Not sure if you need to know it, but my CPU is a i9-7900X.

@rjg Thanks for looking into it, hope it will eventually be resolved! The time measurements I presented weren't the first render after starting the program, but the second/third, anyway I'll do some other tests just to be sure and write them down here. Not sure if you need to know it, but my CPU is a i9-7900X.

slwk1d commented

2020-11-25 21:35:52 +01:00

Did some other tests with the same setup (64x64), same results, this time at the second/fourth render they actually increased from last time, from 2 to 10 seconds.

WITHOUT OPTIX
2nd render: 02:41:37
3rd render: 02:44:59

WITH OPTIX
4th render: 04:32:20
5th render: 04:32:61

Did some other tests with the same setup (64x64), same results, this time at the second/fourth render they actually increased from last time, from 2 to 10 seconds. **WITHOUT OPTIX** 2nd render: 02:41:37 3rd render: 02:44:59 **WITH OPTIX** 4th render: 04:32:20 5th render: 04:32:61

Robert Guetzkow changed title from ~~Rendering time downgrade from 2.90 to 2.91~~ to Cycles CPU rendering performance regression 2.91 and later

2020-11-25 23:38:09 +01:00

Robert Guetzkow commented

2020-11-26 00:03:51 +01:00

Added subscribers: @Jeroen-Bakker, @brecht

Robert Guetzkow commented

2020-11-26 00:03:51 +01:00

I noticed that the project uses adaptive sampling, which may or may not be relevant to the issue at hand.

@brecht @Jeroen-Bakker Have you seen this performance issue during CPU rendering before? I'm a bit hesitant to confirm performance related issues, but this does seem like a regression.

I noticed that the project uses adaptive sampling, which may or may not be relevant to the issue at hand. @brecht @Jeroen-Bakker Have you seen this performance issue during CPU rendering before? I'm a bit hesitant to confirm performance related issues, but this does seem like a regression.

Alaska commented

2020-11-26 05:10:21 +01:00

Added subscriber: @Alaska

Alaska commented

2020-11-26 05:10:21 +01:00

Just thought I'd add:

Rendering with CPU (Ryzen 9 3900X) with tile size 63x64 with OptiX disabled on Linux-5.8.0-3 Debian, these are my results:

2.90.1 release	2.91 release
1:23.34	1:35.75

These results aren't significantly different, like the results show by @rjg and @Slowwkidd, so I thought I may not be experiencing the issue because I'm running Linux. So I retested on Windows (Windows-10-10.0.19041-SP0 64 Bits) with the same CPU and same render settings and these are the results:

2.90.1 release	2.91 release
1:43.43	1:42.94

Based on the small number of tests done so far, it seems the issue doesn't affect Zen 2 CPUs (at the least). Suggesting part of this "regression" may be caused by a change that negatively affects other CPU architectures. Both @rjg and @Slowwkidd are using "older" Intel CPUs. However, this is just speculation.

Just thought I'd add: Rendering with CPU (Ryzen 9 3900X) with tile size 63x64 with OptiX disabled on Linux-5.8.0-3 Debian, these are my results: |2.90.1 release|2.91 release| | -- | -- | |1:23.34|1:35.75| These results aren't significantly different, like the results show by @rjg and @Slowwkidd, so I thought I may not be experiencing the issue because I'm running Linux. So I retested on Windows (`Windows-10-10.0.19041-SP0 64 Bits`) with the same CPU and same render settings and these are the results: |2.90.1 release|2.91 release| | -- | -- | |1:43.43|1:42.94| Based on the small number of tests done so far, it seems the issue doesn't affect Zen 2 CPUs (at the least). Suggesting part of this "regression" may be caused by a change that negatively affects other CPU architectures. Both @rjg and @Slowwkidd are using "older" Intel CPUs. However, this is just speculation.

YAFU commented

2020-11-26 09:22:06 +01:00

Added subscriber: @YAFU

YAFU commented

2020-11-26 09:22:06 +01:00

I'm not sure if the following can influence render times according to Blender version (since it is also something related to CPU work).

I have noticed that apparently Righ s. and Left s meshes fall into some OpenSubdiv 2.8+ still present bug/limitation with Subdivision Surf. Modifier. When you open the .blend file it takes a long time to load. The same happens when you initialize the render. In both cases there is a single threaded process working at 100%. I think it is something related to what has already been described here:
https://developer.blender.org/T58191

If you remove the single face that "closes" the volume in the Righ s. and Left s. the problem I describe seems to go away. Here only Righ s. object in 2.79 .blend file
render bug_mod_2.79.blend
no problem with OpenSubdiv. You open the scene with Blender 2.9 and in Subdivision Surf. Modifier you set Quality to 3 and you will see the performance problem that I explain (it will even freeze blender for a while when you increase Quality setting).

Edit:
Intel i7-3770 here. Ryzen users also have the performance problem that I describe with OpenSubdiv, right?
This is even more evident with this for example:
https://developer.blender.org/T58191#945339

So my intervention in this report is because I'm not sure if OpenSubdiv performance in those cases may have varied a bit between those versions of Blender.

I'm not sure if the following can influence render times according to Blender version (since it is also something related to CPU work). I have noticed that apparently Righ s. and Left s meshes fall into some OpenSubdiv 2.8+ still present bug/limitation with Subdivision Surf. Modifier. When you open the .blend file it takes a long time to load. The same happens when you initialize the render. In both cases there is a single threaded process working at 100%. I think it is something related to what has already been described here: https://developer.blender.org/T58191 If you remove the single face that "closes" the volume in the Righ s. and Left s. the problem I describe seems to go away. Here only Righ s. object in 2.79 .blend file [render bug_mod_2.79.blend](https://archive.blender.org/developer/F9378984/render_bug_mod_2.79.blend) no problem with OpenSubdiv. You open the scene with Blender 2.9 and in Subdivision Surf. Modifier you set Quality to 3 and you will see the performance problem that I explain (it will even freeze blender for a while when you increase Quality setting). Edit: Intel i7-3770 here. Ryzen users also have the performance problem that I describe with OpenSubdiv, right? This is even more evident with this for example: https://developer.blender.org/T58191#945339 So my intervention in this report is because I'm not sure if OpenSubdiv performance in those cases may have varied a bit between those versions of Blender.

slwk1d commented

2020-11-26 09:43:38 +01:00

@YAFU Thanks for pointing out a problem related to subsurf, it's true that regardless of the blender version the file opening and render initializing were slow, but then why in 2.90 the same file renders faster? Might be as @Alaska suggested a regression due to specific CPU architectures?

Brecht Van Lommel commented

2020-11-26 12:13:44 +01:00

Bug reports require as simple as possible .blend files, the intent here is not for Blender developers or triagers to analyze production scenes, we expect the reporter to make an effort to identify what specifically in the .blend file is causing the problem.

Remove as many objects, materials, modifiers, as possible, disable settings, set a low number of samples, and figure out what exactly is causing the issue.

Bug reports require as simple as possible .blend files, the intent here is not for Blender developers or triagers to analyze production scenes, we expect the reporter to make an effort to identify what specifically in the .blend file is causing the problem. Remove as many objects, materials, modifiers, as possible, disable settings, set a low number of samples, and figure out what exactly is causing the issue.

Robert Guetzkow commented

2020-11-26 12:27:37 +01:00

Changed status from 'Needs Triage' to: 'Needs User Info'

slwk1d commented

2020-11-26 14:28:37 +01:00

Sorry for not diving a little bit deeper before, I think I actually found out: the regression in 2.91 regards renders with a large resolution.

Here's is a very simple scene:
Resolution size bug.blend

At 1000x1000 pixels, the scene renders both in 2.90 and 2.91 in around 1 second.
At 3000x3000 pixels, in 2.90 it renders in ~16 seconds, in 2.91 in ~48 seconds.

Important to note that the regression is present in CPU, GPU and CPU+GPU rendering, so it doesn't seem to be connected specifically to CPU issues.

Sorry for not diving a little bit deeper before, I think I actually found out: the regression in 2.91 regards renders with a **large resolution.** Here's is a very simple scene: [Resolution size bug.blend](https://archive.blender.org/developer/F9380933/Resolution_size_bug.blend) At 1000x1000 pixels, the scene renders both in 2.90 and 2.91 in around 1 second. At 3000x3000 pixels, in 2.90 it renders in ~16 seconds, in 2.91 in ~48 seconds. Important to note that the regression is present in CPU, GPU and CPU+GPU rendering, so it doesn't seem to be connected specifically to CPU issues.

Robert Guetzkow commented

2020-11-26 14:40:01 +01:00

That would be #82591 then.

Philipp Oeser commented

2020-11-26 14:49:40 +01:00

Added subscriber: @lichtwerk

Philipp Oeser commented

2020-11-26 14:49:40 +01:00

Is this the same as {#82591}?

Robert Guetzkow commented

2020-11-26 14:51:31 +01:00

Closed as duplicate of #82591

Robert Guetzkow closed this issue

2020-11-26 14:51:31 +01:00

Robert Guetzkow commented

2020-11-26 14:53:31 +01:00

I didn't expect that this would have such an impact on a relatively small output resolution though. The 1754x2481 render of the example project is not nearly as gigantic as the case discussed in the #82591.

slwk1d commented

2020-11-26 14:56:02 +01:00

This comment was removed by @Slowwkidd

*This comment was removed by @Slowwkidd*

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Cycles CPU rendering performance regression 2.91 and later #83011