Commit Graph

29 Commits

Author SHA1 Message Date
feeb8310c8 Cleanup: format 2022-06-30 12:14:23 +10:00
Jeroen Bakker
66f826ae85 Benchmark: Add eevee viewport playback tests.
This commit adds the ability to test Eevee viewport playback performance tests.

Tests should be placed in `lib/benchmarks/eevee/*/*.blend`. {rBL62962} added
initial test files. See https://wiki.blender.org/wiki/Tools/Tests/Performance how
to set it up.

To record the playback performance the test start the viewport playback, and adds
a post frame change handler.

This handler will go over the next steps:

* Ensures the viewport is set to rendered mode.
* Wait for shaders to be compiled. Utilizes `bpy.app.is_job_running` function when
  available (v3.3) to wait for shader compilation to finish. When not available will wait
  for one minute.
* Draw several warmup frames
* Record for 10 seconds tracking the number of frames drawn and performance counters.
* When ready print the result to the console. The results will be extracted when the
  benchmark has run.

## Example report
```
                                         master               v3.0                 v3.1                 v3.2
T88219                                   0.0860s              0.0744s              0.0744s              0.0851s
blender290-fox                           1.3056s              0.8744s              0.7994s              1.2809s
```

{F13232387}

Reviewed By: brecht, fclem

Maniphest Tasks: T99136

Differential Revision: https://developer.blender.org/D15302
2022-06-29 07:55:51 +02:00
bba757ef81 Cleanup: various minor changes
- Add missing doxy-section for Apply Parent Inverse Operator
- Use identity for None comparison in Python.
- Remove newline from operator doc-strings.
- Use '*' prefix multi-line C comment blocks.
- Separate filenames from doc-strings.
- Remove break after return.
2022-04-24 13:41:03 +10:00
65a1fcdaf7 Cleanup: run autopep8 on tests/performance/benchmark
This file was skipped by source/tools/utils/autopep8_clean.py
since it doesn't have a .py extension, running the autopep8 tool
recursively detects Python scripts without extensions.
2022-04-20 17:07:46 +10:00
3035235def Cleanup: run autopep8 on tests/ 2022-04-19 15:25:55 +10:00
c434782e3a File headers: SPDX License migration
Use a shorter/simpler license convention, stops the header taking so
much space.

Follow the SPDX license specification: https://spdx.org/licenses

- C/C++/objc/objc++
- Python
- Shell Scripts
- CMake, GNUmakefile

While most of the source tree has been included

- `./extern/` was left out.
- `./intern/cycles` & `./intern/atomic` are also excluded because they
  use different header conventions.

doc/license/SPDX-license-identifiers.txt has been added to list SPDX all
used identifiers.

See P2788 for the script that automated these edits.

Reviewed By: brecht, mont29, sergey

Ref D14069
2022-02-11 09:14:36 +11:00
8ae6995e98 Tests: show more precise numbers in benchmark charts 2022-01-14 15:17:22 +01:00
7d5ef64bfb Cleanup: fix typos in comments and docs
Contributed by luzpaz.

Differential Revision: https://developer.blender.org/D13264
2021-11-19 12:46:49 +01:00
1b55b911f2 Merge branch 'blender-v3.0-release' 2021-11-12 20:04:05 +01:00
71131b4969 Tests: disable audio system for performance tests 2021-11-12 20:03:46 +01:00
4648c4990c Merge branch 'blender-v3.0-release' 2021-11-09 13:08:36 -06:00
7c25399576 tests/benchmarks: Fix operation on windows
The test script did not work on windows
since it had some trouble importing the
api module on the blender side of things.

turning the file path to the module into
a raw string literal sidesteps the
backslash issue in the path.

Differential Revision: https://developer.blender.org/D13163

Reviewed by: brecht
2021-11-09 09:58:48 -07:00
27b37517f8 Cleanup: use static sets 2021-11-08 17:00:36 +11:00
c516659b5e Cleanup: use static sets 2021-11-08 14:52:08 +11:00
fc36772b06 Tests: minor updates to benchmark script for running on buildbot
* graph command accepts folder of json files as input
* reset command clears log files
2021-10-28 22:41:40 +02:00
335f40ebfa Tests: include device type in benchmark graph labels 2021-10-06 14:21:26 +02:00
cc653c9b02 Fix potential render tests error with invalid utf-8 characters
In general should not happen, but better to report the actual error instead
of the Python test code failing.
2021-09-28 20:50:38 +02:00
0803119725 Cycles: merge of cycles-x branch, a major update to the renderer
This includes much improved GPU rendering performance, viewport interactivity,
new shadow catcher, revamped sampling settings, subsurface scattering anisotropy,
new GPU volume sampling, improved PMJ sampling pattern, and more.

Some features have also been removed or changed, breaking backwards compatibility.
Including the removal of the OpenCL backend, for which alternatives are under
development.

Release notes and code docs:
https://wiki.blender.org/wiki/Reference/Release_Notes/3.0/Cycles
https://wiki.blender.org/wiki/Source/Render/Cycles

Credits:
* Sergey Sharybin
* Brecht Van Lommel
* Patrick Mours (OptiX backend)
* Christophe Hery (subsurface scattering anisotropy)
* William Leeson (PMJ sampling pattern)
* Alaska (various fixes and tweaks)
* Thomas Dinges (various fixes)

For the full commit history, see the cycles-x branch. This squashes together
all the changes since intermediate changes would often fail building or tests.

Ref T87839, T87837, T87836
Fixes T90734, T89353, T80267, T80267, T77185, T69800
2021-09-21 14:55:54 +02:00
e43ecca016 Tests: measure time per frame in animation performance benchmark
Instead of a fixed number of frames, so that benchmarking takes about the
same time on any machine.
2021-09-20 19:03:22 +02:00
42215d7cb8 Tests: more graceful handling of keyboard interrupting benchmarks
Leave current test result unchanged and stop executing immediately,
so it can be continued.
2021-09-10 16:48:26 +02:00
5bfc3a3421 Fix error running benchmark script with environment variables for builds
Ref D12434
2021-09-08 19:58:27 +02:00
6fc94d1848 Tests: updates for performance benchmarking
* Make "run" command (re-)run all tests, add "update" command to only
  run queued and outdated tests equivalent to the old "run" command.
* Support specifying environment variables for revisions, to easily
  compare multiple parameter values.
* Better sorting of revisions in graph.
2021-09-08 16:40:58 +02:00
214e4aac97 Fix Python error in ./benchmark init after recent changes 2021-08-19 17:45:37 +02:00
22b03e1c68 Fix Python error in benchmark executable detection after recent changes 2021-07-26 18:04:40 +02:00
320f34af86 Tests: continue running benchmarks if some tests fail to build or run
Convenient when testing many revisions where some might be broken.
2021-07-22 16:35:00 +02:00
19b597c55d Tests: improve finding of Blender executables in benchmarking
* Allow specifying a folder and automatically setting the proper executable
  name depending on the operating system
* Use executables from configs for listing devices instead of a blender
  command being available
2021-07-22 16:34:57 +02:00
37a5ff4a84 Tests: support graphing peak memory in Cycles performance tests
The general graphing mechanism will create one graph for each output
variable. So it's not limited to time and memory, but that is what the
Cycles tests now output.
2021-07-14 16:12:10 +02:00
432bfbf7a3 Cleanup: pep8 2021-07-06 12:05:27 +10:00
dc3f46d96b Tests: performance testing framework
These are scripts for benchmarking Blender features on real-world .blend
files. They were originally written for benchmarking Cycles performance, and
were made generic so they can be used for more Blender features.

The benchmarks can be run locally by developers. But the plan is to also run
these as part of continuous integration to track performance over time.

Currently there are tests for Cycles rendering and .blend file loading.

Documentation:
https://wiki.blender.org/wiki/Tools/Tests/Performance

Main features:
* User created configurations to quickly run, re-run and analyze a selected
  subset of tests.
* Supports both benchmarking with existing builds, and automatic building of
  specified git commits, tags and branches.
* Generate HTML page with bar and line graphs from test results.
* Controlled using simple command line tool.
* For writing tests, convenient abstraction to run a Python function in Blender
  with arguments and return value.

Ref T74730

Differential Revision: https://developer.blender.org/D11662
2021-07-05 12:32:32 +02:00