archive/blender-benchmark-bundle
Archived
1
0
Fork 0
Code Issues 12 Pull Requests Projects Releases Wiki Activity

Blender Opendata .blend files can be modified before benchmarking to produce unreliable results. #76290

New Issue
Closed
opened 2020-05-01 09:38:42 +02:00 by Alaska · 4 comments
Alaska commented 2020-05-01 09:38:42 +02:00
Copy Link

System Information:
Operating system: Linux

OpenData version:
Broken: OpenData 2.0.5

Short description of error:
The local .blend files stored for the OpenData benchmarking tool can be modified before the benchmark is run. This can be done to create unfair tests resulting in the opendata website becoming a unreliable source if one user was to test the scene over and over again or if multiple users tested with modified .blend files.

I don't know if the OpenData website has a system in place to stop users from uploading outlier results simply down to the fact I didn't want to post my own un-reliable results.

An option to fix this is to run a checksum once the user has selected the scenes they would like to benchmark. If the checksum does not match, re-download the .blend file. Following that, run another checksum before loading the file to benchmark. And if it doesn't match then, maybe skip the benchmark and report back to the user with a "Render failed: Corrupt .blend".

This also has the benefit of the OpenData benchmark tool being able to detect corrupted downloads (assuming a system doesn't exist already). So maybe this checksum system could also be applied to the Blender executable download aswell.

I understand it's basically impossible to stop someone from maliciously posting invalid results, and sometimes the results aren't even meant to be maliciously (E.G. All the users modifying the original OpenData benchmark so they could test RTX GPUs in 2.79 or tests with overclocked systems or systems that thermal throttle) but implementing this could be an extra step to stop some users from going out of their way to produce invalid results.

Steps to reproduce the error:

  1. Download the Blender OpenData benchmark tool and tell it to download a version of Blender and some benchmark scenes.
  2. Find the location where the .blend files for the benchmark are stored (depends on OS).
  3. Open one of the benchmark files in a normal version of Blender.
  4. Adjust a setting (E.G. Reduce the sample count) and save the file.
  5. In the OpenData benchmark application start the benchmark with whatever device you want. Results will no longer be consistent with what's expected from the hardware because the tests are now different.
**System Information:** Operating system: Linux **OpenData version:** Broken: OpenData 2.0.5 **Short description of error:** The local .blend files stored for the OpenData benchmarking tool can be modified before the benchmark is run. This can be done to create unfair tests resulting in the opendata website becoming a unreliable source if one user was to test the scene over and over again or if multiple users tested with modified .blend files. I don't know if the OpenData website has a system in place to stop users from uploading outlier results simply down to the fact I didn't want to post my own un-reliable results. An option to fix this is to run a checksum once the user has selected the scenes they would like to benchmark. If the checksum does not match, re-download the .blend file. Following that, run another checksum before loading the file to benchmark. And if it doesn't match then, maybe skip the benchmark and report back to the user with a "Render failed: Corrupt .blend". This also has the benefit of the OpenData benchmark tool being able to detect corrupted downloads (assuming a system doesn't exist already). So maybe this checksum system could also be applied to the Blender executable download aswell. I understand it's basically impossible to stop someone from maliciously posting invalid results, and sometimes the results aren't even meant to be maliciously (E.G. All the users modifying the original OpenData benchmark so they could test RTX GPUs in 2.79 or tests with overclocked systems or systems that thermal throttle) but implementing this could be an extra step to stop some users from going out of their way to produce invalid results. **Steps to reproduce the error:** 1. Download the Blender OpenData benchmark tool and tell it to download a version of Blender and some benchmark scenes. 2. Find the location where the .blend files for the benchmark are stored (depends on OS). 3. Open one of the benchmark files in a normal version of Blender. 4. Adjust a setting (E.G. Reduce the sample count) and save the file. 5. In the OpenData benchmark application start the benchmark with whatever device you want. Results will no longer be consistent with what's expected from the hardware because the tests are now different.
Alaska commented 2020-05-01 09:38:42 +02:00
Author
Copy Link

Added subscriber: @Alaska

Added subscriber: @Alaska
Brecht Van Lommel commented 2020-05-01 15:36:27 +02:00
Copy Link

Added subscriber: @brecht

Added subscriber: @brecht
Brecht Van Lommel commented 2020-05-01 15:36:27 +02:00
Copy Link

Changed status from 'Needs Triage' to: 'Archived'

Changed status from 'Needs Triage' to: 'Archived'
Brecht Van Lommel self-assigned this 2020-05-01 15:36:27 +02:00
Brecht Van Lommel commented 2020-05-01 15:36:27 +02:00
Copy Link

There are systems in place to remove outliers, this is not considered a bug.

There are systems in place to remove outliers, this is not considered a bug.
This repo is archived. You cannot comment on issues.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Priority::Normal
Status::Archived
Status::Confirmed
Status::Duplicate
Status::Needs Triage
Status::Resolved
Type::Bug
Type::Design
Type::Report
Type::To Do
legacy module
Rendering & Cycles
legacy project
1.0.0-beta.2
legacy project
Blender Benchmark
legacy project
Cycles
No Label
Priority::Normal
Status::Archived
Type::Report
legacy project
Blender Benchmark
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
blender-admin
No Assignees Brecht Van Lommel
2 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: archive/blender-benchmark-bundle#76290
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?