RenderTest: GPU Render Tests Pass Silently #117629

Merged
Jeroen Bakker merged 2 commits from Jeroen-Bakker/blender:render-tests/fail-silently into main 2024-01-29 15:39:25 +01:00
Member

Add silently fail option to GPU based render tests. This is a pre-requisite to enable
render tests on the buildbot. By default these render tests will pass silently.

  • Test will pass when using the --pass-silently arguments.
  • Only crashes will be reported as failed tests.
  • To find out failing test, review the test reports.

WITH_GPU_RENDER_TESTS_SILENT compile option can be used to let tests pass (default)
or fail (default for developers).

Although some tests fail, they still passed. In the generated render report,
the silently passed failures are correctly reported to be failures.

WITH_GPU_RENDER_TESTS_SILENT=On

        Start 148: eevee_next_bsdf
148/435 Test #148: eevee_next_bsdf .................................................   Passed   23.29 sec
        Start 149: eevee_next_camera
149/435 Test #149: eevee_next_camera ...............................................   Passed   10.77 sec
        Start 150: eevee_next_displacement
150/435 Test #150: eevee_next_displacement .........................................   Passed   13.80 sec
        Start 151: eevee_next_grease_pencil
151/435 Test #151: eevee_next_grease_pencil ........................................***Failed    1.70 sec

WITH_GPU_RENDER_TESTS_SILENT=Off

     Start 148: eevee_next_bsdf
 1/20 Test #148: eevee_next_bsdf ..................***Failed   23.55 sec
      Start 149: eevee_next_camera
 2/20 Test #149: eevee_next_camera ................   Passed   10.76 sec
      Start 150: eevee_next_displacement
 3/20 Test #150: eevee_next_displacement ..........***Failed   13.87 sec
      Start 151: eevee_next_grease_pencil
 4/20 Test #151: eevee_next_grease_pencil .........***Failed    1.70 sec

There is one crashing test (grease pencil) that needs to be fixed, before render tests can be enabled on the buildbot.
Fixing the crashing test and enabling on the build bot will be done in separate commits. Start 148: eevee_next_bsdf

Add silently fail option to GPU based render tests. This is a pre-requisite to enable render tests on the buildbot. By default these render tests will pass silently. * Test will pass when using the `--pass-silently` arguments. * Only crashes will be reported as failed tests. * To find out failing test, review the test reports. `WITH_GPU_RENDER_TESTS_SILENT` compile option can be used to let tests pass (default) or fail (default for developers). Although some tests fail, they still passed. In the generated render report, the silently passed failures are correctly reported to be failures. *WITH_GPU_RENDER_TESTS_SILENT=On* ``` Start 148: eevee_next_bsdf 148/435 Test #148: eevee_next_bsdf ................................................. Passed 23.29 sec Start 149: eevee_next_camera 149/435 Test #149: eevee_next_camera ............................................... Passed 10.77 sec Start 150: eevee_next_displacement 150/435 Test #150: eevee_next_displacement ......................................... Passed 13.80 sec Start 151: eevee_next_grease_pencil 151/435 Test #151: eevee_next_grease_pencil ........................................***Failed 1.70 sec ``` *WITH_GPU_RENDER_TESTS_SILENT=Off* ``` Start 148: eevee_next_bsdf 1/20 Test #148: eevee_next_bsdf ..................***Failed 23.55 sec Start 149: eevee_next_camera 2/20 Test #149: eevee_next_camera ................ Passed 10.76 sec Start 150: eevee_next_displacement 3/20 Test #150: eevee_next_displacement ..........***Failed 13.87 sec Start 151: eevee_next_grease_pencil 4/20 Test #151: eevee_next_grease_pencil .........***Failed 1.70 sec ``` > There is one crashing test (grease pencil) that needs to be fixed, before render tests can be enabled on the buildbot. > Fixing the crashing test and enabling on the build bot will be done in separate commits. Start 148: eevee_next_bsdf
Jeroen Bakker added this to the 4.1 milestone 2024-01-29 13:21:46 +01:00
Jeroen Bakker added the
Module
Platforms, Builds & Tests
label 2024-01-29 13:21:46 +01:00
Jeroen Bakker self-assigned this 2024-01-29 13:21:47 +01:00
Jeroen Bakker added 1 commit 2024-01-29 13:21:56 +01:00
Add silently fail option to GPU based render tests. This is done to enable
render tests on the buildbot. By default these render tests will pass silently.

* Test will pass when using the `--pass-silently` arguments.
* Only crashes will be reported as failed tests.
* To find out failing test, review the test reports.

* [ ] Add compile option to make `pass-silently` optional. So we can enable it only on the buildbot
Jeroen Bakker added this to the Platforms, Builds Tests & Devices project 2024-01-29 13:23:14 +01:00
Jeroen Bakker requested review from Brecht Van Lommel 2024-01-29 13:23:41 +01:00

Is there some console output for failed tests, or is it just the HTML page that shows it?

Is there some console output for failed tests, or is it just the HTML page that shows it?
Jeroen Bakker added 1 commit 2024-01-29 13:47:25 +01:00
Add compile directive
All checks were successful
buildbot/vexp-code-patch-darwin-x86_64 Build done.
buildbot/vexp-code-patch-linux-x86_64 Build done.
buildbot/vexp-code-patch-lint Build done.
buildbot/vexp-code-patch-windows-amd64 Build done.
buildbot/vexp-code-patch-darwin-arm64 Build done.
buildbot/vexp-code-patch-coordinator Build done.
f56daf791e
Jeroen Bakker changed title from WIP: RenderTest: GPU Render Tests Pass Silently to RenderTest: GPU Render Tests Pass Silently 2024-01-29 13:52:46 +01:00
Author
Member

So far I only saw it in the HTML report and /Testing/Temporary/LastTest.log

So far I only saw it in the HTML report and `/Testing/Temporary/LastTest.log`
Brecht Van Lommel approved these changes 2024-01-29 14:19:53 +01:00

Ok. Maybe I'll add support for uploading the test files unzipped, so it's easy to check online.

Ok. Maybe I'll add support for uploading the test files unzipped, so it's easy to check online.
Author
Member

@blender-bot build

@blender-bot build
Jeroen Bakker merged commit 324ef0cbf4 into main 2024-01-29 15:39:25 +01:00
Sign in to join this conversation.
No reviewers
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset System
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Asset Browser Project
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#117629
No description provided.