RenderTest: GPU Render Tests Pass Silently #117629

Jeroen Bakker · 2024-01-29T13:21:46+01:00

Jeroen Bakker commented

2024-01-29 13:21:46 +01:00

Add silently fail option to GPU based render tests. This is a pre-requisite to enable
render tests on the buildbot. By default these render tests will pass silently.

Test will pass when using the --pass-silently arguments.
Only crashes will be reported as failed tests.
To find out failing test, review the test reports.

WITH_GPU_RENDER_TESTS_SILENT compile option can be used to let tests pass (default)
or fail (default for developers).

Although some tests fail, they still passed. In the generated render report,
the silently passed failures are correctly reported to be failures.

WITH_GPU_RENDER_TESTS_SILENT=On

        Start 148: eevee_next_bsdf
148/435 Test #148: eevee_next_bsdf .................................................   Passed   23.29 sec
        Start 149: eevee_next_camera
149/435 Test #149: eevee_next_camera ...............................................   Passed   10.77 sec
        Start 150: eevee_next_displacement
150/435 Test #150: eevee_next_displacement .........................................   Passed   13.80 sec
        Start 151: eevee_next_grease_pencil
151/435 Test #151: eevee_next_grease_pencil ........................................***Failed    1.70 sec

WITH_GPU_RENDER_TESTS_SILENT=Off

     Start 148: eevee_next_bsdf
 1/20 Test #148: eevee_next_bsdf ..................***Failed   23.55 sec
      Start 149: eevee_next_camera
 2/20 Test #149: eevee_next_camera ................   Passed   10.76 sec
      Start 150: eevee_next_displacement
 3/20 Test #150: eevee_next_displacement ..........***Failed   13.87 sec
      Start 151: eevee_next_grease_pencil
 4/20 Test #151: eevee_next_grease_pencil .........***Failed    1.70 sec

There is one crashing test (grease pencil) that needs to be fixed, before render tests can be enabled on the buildbot.
Fixing the crashing test and enabling on the build bot will be done in separate commits. Start 148: eevee_next_bsdf

Add silently fail option to GPU based render tests. This is a pre-requisite to enable render tests on the buildbot. By default these render tests will pass silently. * Test will pass when using the `--pass-silently` arguments. * Only crashes will be reported as failed tests. * To find out failing test, review the test reports. `WITH_GPU_RENDER_TESTS_SILENT` compile option can be used to let tests pass (default) or fail (default for developers). Although some tests fail, they still passed. In the generated render report, the silently passed failures are correctly reported to be failures. *WITH_GPU_RENDER_TESTS_SILENT=On* ``` Start 148: eevee_next_bsdf 148/435 Test #148: eevee_next_bsdf ................................................. Passed 23.29 sec Start 149: eevee_next_camera 149/435 Test #149: eevee_next_camera ............................................... Passed 10.77 sec Start 150: eevee_next_displacement 150/435 Test #150: eevee_next_displacement ......................................... Passed 13.80 sec Start 151: eevee_next_grease_pencil 151/435 Test #151: eevee_next_grease_pencil ........................................***Failed 1.70 sec ``` *WITH_GPU_RENDER_TESTS_SILENT=Off* ``` Start 148: eevee_next_bsdf 1/20 Test #148: eevee_next_bsdf ..................***Failed 23.55 sec Start 149: eevee_next_camera 2/20 Test #149: eevee_next_camera ................ Passed 10.76 sec Start 150: eevee_next_displacement 3/20 Test #150: eevee_next_displacement ..........***Failed 13.87 sec Start 151: eevee_next_grease_pencil 4/20 Test #151: eevee_next_grease_pencil .........***Failed 1.70 sec ``` > There is one crashing test (grease pencil) that needs to be fixed, before render tests can be enabled on the buildbot. > Fixing the crashing test and enabling on the build bot will be done in separate commits. Start 148: eevee_next_bsdf

Jeroen Bakker added this to the 4.1 milestone 2024-01-29 13:21:46 +01:00

Jeroen Bakker added the

Module

Platforms, Builds & Tests

label 2024-01-29 13:21:46 +01:00

Jeroen Bakker self-assigned this 2024-01-29 13:21:47 +01:00

Jeroen Bakker added 1 commit 2024-01-29 13:21:56 +01:00

RenderTest: GPU Render Tests Pass Silently cc3b2b83c5

Add silently fail option to GPU based render tests. This is done to enable
render tests on the buildbot. By default these render tests will pass silently.

* Test will pass when using the `--pass-silently` arguments.
* Only crashes will be reported as failed tests.
* To find out failing test, review the test reports.

* [ ] Add compile option to make `pass-silently` optional. So we can enable it only on the buildbot

Jeroen Bakker added this to the Platforms, Builds Tests & Devices project 2024-01-29 13:23:14 +01:00

Jeroen Bakker requested review from Brecht Van Lommel 2024-01-29 13:23:41 +01:00

Brecht Van Lommel commented

2024-01-29 13:27:32 +01:00

Is there some console output for failed tests, or is it just the HTML page that shows it?

Jeroen Bakker added 1 commit 2024-01-29 13:47:25 +01:00

Add compile directive

buildbot/vexp-code-patch-darwin-x86_64 Build done.

Details

buildbot/vexp-code-patch-linux-x86_64 Build done.

Details

buildbot/vexp-code-patch-lint Build done.