Hard to reproduce crash while rendering animation #77734

New Issue

Vincent Blankfield · 2020-06-10T19:06:42+02:00

2020-06-10 19:06:42 +02:00

System Information
Operating system: Windows-10-10.0.14393-SP0 64 Bits
Graphics card: GeForce GTX 1060 6GB/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 442.19

Blender Version
Broken: version: 2.90.0 Alpha current master, 2.83.0, branch: master, commit date: 2020-06-03 14:38, hash: 211b6c29f7
Worked: unknown

Short description of error
Both 2.83 and 2.90 sometimes crash at random frame during animation render. The bug is hard to catch reliably, but it seems to happen more often wtih some render settings. In particular I couldn't reproduce it without "presistent images" checked. Also it seems to be sensitive to frame count, render sample count, output resolution and tile size. Tested only with Cycles. Happens both with GPU and CPU render, with CPU much less frequently. Tested only on one Windows machine. The crash only happens on release builds (2.83 - official installer, 2.90 - self built with more or less standard settings). The issue is particularly hard to catch as it dissappears as soon as any code is altered or a breakpoint/logpoint is placed in the suspicious code. Changing build optimizations also breaks reproducibility.

The crash happens at external_engine.h:560:

  GHashIterator pa_iter;
  GHASH_ITER (pa_iter, re->parts) {
    RenderPart *pa = BLI_ghashIterator_getValue(&pa_iter); *HERE* pa_iter.curEntry is 0xFFFFFFFFFFFFFFEF

Stack trace:

blender.exe!RE_engine_get_current_tiles(Render * re, int * r_total_tiles, bool * r_needs_free) Line 560 (.\blender\source\blender\render\intern\source\external_engine.c:560)
blender.exe!draw_render_info(const bContext * C, Scene * scene, Image * ima, ARegion * region, float zoomx, float zoomy) Line 104 (.\blender\source\blender\editors\space_image\image_draw.c:104)
blender.exe!draw_image_main(const bContext * C, ARegion * region) Line 1037 (.\blender\source\blender\editors\space_image\image_draw.c:1037)
blender.exe!image_main_region_draw(const bContext * C, ARegion * region) Line 677 (.\blender\source\blender\editors\space_image\space_image.c:677)
blender.exe!ED_region_do_draw(bContext * C, ARegion * region) Line 543 (.\blender\source\blender\editors\screen\area.c:543)
blender.exe!wm_draw_window_offscreen(bContext * C, wmWindow * win, bool stereo) Line 689 (.\blender\source\blender\windowmanager\intern\wm_draw.c:689)
blender.exe!wm_draw_window(bContext * C, wmWindow * win) Line 812 (.\blender\source\blender\windowmanager\intern\wm_draw.c:812)
blender.exe!wm_draw_update(bContext * C) Line 1013 (.\blender\source\blender\windowmanager\intern\wm_draw.c:1013)
blender.exe!WM_main(bContext * C) Line 482 (.\blender\source\blender\windowmanager\intern\wm.c:482)
blender.exe!main(int argc, const unsigned char * * UNUSED_argv_c) Line 530 (.\blender\source\creator\creator.c:530)
[Inline Frame] blender.exe!invoke_main() Line 78 (d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78)
blender.exe!__scrt_common_main_seh() Line 288 (d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
kernel32.dll!00007fff3eb684d4() (Unknown Source:0)
ntdll.dll!00007fff3f9ae871() (Unknown Source:0)

Exact steps for others to reproduce the error

Open blender 2.83 or 2.90 (default blend file)
Check "presistent images"
Minimize sample count, output resolution and tile size for faster render, but make the resolution at least 4xtile_size.
Render animation
If id doesn't crah alter the settings from step 3 and repeat. Changing frame count may also help.

Or use one of the attached files:
crash-fast-unreliable.blend
crash-unsimplified-more-reliable.blend

hat_2.83_debug_logs.zip

**System Information** Operating system: Windows-10-10.0.14393-SP0 64 Bits Graphics card: GeForce GTX 1060 6GB/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 442.19 **Blender Version** Broken: version: 2.90.0 Alpha current master, 2.83.0, branch: master, commit date: 2020-06-03 14:38, hash: `211b6c29f7` Worked: unknown **Short description of error** Both `2.83` and `2.90` sometimes crash at random frame during animation render. The bug is hard to catch reliably, but it seems to happen more often wtih some render settings. In particular I couldn't reproduce it without "presistent images" checked. Also it seems to be sensitive to frame count, render sample count, output resolution and tile size. Tested only with Cycles. Happens both with GPU and CPU render, with CPU much less frequently. Tested only on one Windows machine. The crash only happens on release builds (`2.83` - official installer, `2.90` - self built with more or less standard settings). The issue is particularly hard to catch as it dissappears as soon as any code is altered or a breakpoint/logpoint is placed in the suspicious code. Changing build optimizations also breaks reproducibility. The crash happens at `external_engine.h:560`: ``` GHashIterator pa_iter; GHASH_ITER (pa_iter, re->parts) { RenderPart *pa = BLI_ghashIterator_getValue(&pa_iter); *HERE* pa_iter.curEntry is 0xFFFFFFFFFFFFFFEF ``` Stack trace: ``` blender.exe!RE_engine_get_current_tiles(Render * re, int * r_total_tiles, bool * r_needs_free) Line 560 (.\blender\source\blender\render\intern\source\external_engine.c:560) blender.exe!draw_render_info(const bContext * C, Scene * scene, Image * ima, ARegion * region, float zoomx, float zoomy) Line 104 (.\blender\source\blender\editors\space_image\image_draw.c:104) blender.exe!draw_image_main(const bContext * C, ARegion * region) Line 1037 (.\blender\source\blender\editors\space_image\image_draw.c:1037) blender.exe!image_main_region_draw(const bContext * C, ARegion * region) Line 677 (.\blender\source\blender\editors\space_image\space_image.c:677) blender.exe!ED_region_do_draw(bContext * C, ARegion * region) Line 543 (.\blender\source\blender\editors\screen\area.c:543) blender.exe!wm_draw_window_offscreen(bContext * C, wmWindow * win, bool stereo) Line 689 (.\blender\source\blender\windowmanager\intern\wm_draw.c:689) blender.exe!wm_draw_window(bContext * C, wmWindow * win) Line 812 (.\blender\source\blender\windowmanager\intern\wm_draw.c:812) blender.exe!wm_draw_update(bContext * C) Line 1013 (.\blender\source\blender\windowmanager\intern\wm_draw.c:1013) blender.exe!WM_main(bContext * C) Line 482 (.\blender\source\blender\windowmanager\intern\wm.c:482) blender.exe!main(int argc, const unsigned char * * UNUSED_argv_c) Line 530 (.\blender\source\creator\creator.c:530) [Inline Frame] blender.exe!invoke_main() Line 78 (d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78) blender.exe!__scrt_common_main_seh() Line 288 (d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288) kernel32.dll!00007fff3eb684d4() (Unknown Source:0) ntdll.dll!00007fff3f9ae871() (Unknown Source:0) ``` **Exact steps for others to reproduce the error** 1. Open blender 2.83 or 2.90 (default blend file) 2. Check "presistent images" 3. Minimize sample count, output resolution and tile size for faster render, but make the resolution at least 4xtile_size. 4. Render animation 5. If id doesn't crah alter the settings from step 3 and repeat. Changing frame count may also help. Or use one of the attached files: [crash-fast-unreliable.blend](https://archive.blender.org/developer/F8606386/crash-fast-unreliable.blend) [crash-unsimplified-more-reliable.blend](https://archive.blender.org/developer/F8606390/crash-unsimplified-more-reliable.blend) [hat_2.83_debug_logs.zip](https://archive.blender.org/developer/F8606397/hat_2.83_debug_logs.zip)

Vincent Blankfield commented

2020-06-10 19:06:42 +02:00

Added subscriber: @VincentBlankfield

blender-admin commented

2020-06-10 19:06:42 +02:00

#77918 was marked as duplicate of this issue

Vincent Blankfield commented

2020-06-11 19:41:54 +02:00

I suddenly realized a should-be-obvious way to make the crash more reliably reproducible (not 100% though):

diff --git a/source/blender/render/intern/source/initrender.c b/source/blender/render/intern/source/initrender.c
index 2e9f30397db..eb8e2b1a05e 100644
--- a/source/blender/render/intern/source/initrender.c
+++ b/source/blender/render/intern/source/initrender.c
@@ -327,6 +327,16 @@ void RE_parts_init(Render *re)
       pa->rectx = rectx;
       pa->recty = recty;

+      /*--- Make a part intentionally invalid for a longer while ---*/
+      RenderPart *pa1 = NULL;
+      BLI_ghash_insert(re->parts, &pa->disprect, pa1);
+      for (int i = 0; i < 1000; i++) {
+        printf(".");
+      }
+      printf("\n");
+      BLI_ghash_remove(re->parts, &pa->disprect, NULL, NULL);
+      /*------------------------------------------------------------*/
+
       BLI_ghash_insert(re->parts, &pa->disprect, pa);
       re->i.totpart++;
     }

Same crash, same place, crashes on the first-second frame with the test file, based on the default scene, with following settings:

engine: Cycles
render sample count: 1 sample
resolution: 128x128
tile size: 8px
presistent images: checked

With other test files crashes less likely. It depends on the rendering speed. Without the "presistent images" checked I didn't manage to get a crash.

The proposed fix still works, but I lack the big picture of how the threads interact, so there may be a problem somwhere else.

I suddenly realized a should-be-obvious way to make the crash more reliably reproducible (not 100% though): ``` diff --git a/source/blender/render/intern/source/initrender.c b/source/blender/render/intern/source/initrender.c index 2e9f30397db..eb8e2b1a05e 100644 --- a/source/blender/render/intern/source/initrender.c +++ b/source/blender/render/intern/source/initrender.c @@ -327,6 +327,16 @@ void RE_parts_init(Render *re) pa->rectx = rectx; pa->recty = recty; + /*--- Make a part intentionally invalid for a longer while ---*/ + RenderPart *pa1 = NULL; + BLI_ghash_insert(re->parts, &pa->disprect, pa1); + for (int i = 0; i < 1000; i++) { + printf("."); + } + printf("\n"); + BLI_ghash_remove(re->parts, &pa->disprect, NULL, NULL); + /*------------------------------------------------------------*/ + BLI_ghash_insert(re->parts, &pa->disprect, pa); re->i.totpart++; } ``` Same crash, same place, crashes on the first-second frame with the test file, based on the default scene, with following settings: ``` engine: Cycles render sample count: 1 sample resolution: 128x128 tile size: 8px presistent images: checked ``` With other test files crashes less likely. It depends on the rendering speed. Without the "presistent images" checked I didn't manage to get a crash. The proposed fix still works, but I lack the big picture of how the threads interact, so there may be a problem somwhere else.

Germano Cavalcante commented

2020-06-18 16:05:49 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'