Improve device selection in benchmark #73325

New Issue

Patrick Mours · 2020-01-22T18:57:30+01:00

Patrick Mours commented

2020-01-22 18:57:30 +01:00

Currently the device selection in the benchmark only allows you to select a single GPU to benchmark on. Yet for CPUs it is possible to get results with multiple CPUs in the system (as evident by the 2x AMD/Intel ... entries on the Open Data website).
It would be fairer to also allow multi-GPU configurations (but e.g. restrict them to only GPUs of the same type).

On a similar note: The UI version of the benchmark currently preselects a device for benchmarking and only shows them all in a drop down. I'd suggest always showing all available devices and make a selection with radio buttons instead, so it is e.g. obvious at a glance that you have two API options for RTX GPUs (CUDA and OptiX).

Currently the device selection in the benchmark only allows you to select a single GPU to benchmark on. Yet for CPUs it is possible to get results with multiple CPUs in the system (as evident by the `2x AMD/Intel ...` entries on the Open Data website). It would be fairer to also allow multi-GPU configurations (but e.g. restrict them to only GPUs of the same type). On a similar note: The UI version of the benchmark currently preselects a device for benchmarking and only shows them all in a drop down. I'd suggest always showing all available devices and make a selection with radio buttons instead, so it is e.g. obvious at a glance that you have two API options for RTX GPUs (CUDA and OptiX).

Patrick Mours commented

2020-01-22 18:57:30 +01:00

Added subscriber: @pmoursnv

Sem Mulder commented

2020-01-27 11:38:54 +01:00

Added subscriber: @SemMulder

Sem Mulder commented

2020-01-27 11:38:54 +01:00

I'm open to allowing multiple GPUs. We do need to figure out some stuff though:

Do we only allow GPUs of the same kind, as you mentioned?
Do we allow the user to pick the number of cards? (if so, how?)

radio buttons instead

I agree.

I'm open to allowing multiple GPUs. We do need to figure out some stuff though: - Do we only allow GPUs of the same kind, as you mentioned? - Do we allow the user to pick the number of cards? (if so, how?) > radio buttons instead I agree.

Patrick Mours commented

2020-01-27 13:35:29 +01:00

Restricting to GPUs of the same kind makes for cleaner benchmark results I think. Otherwise the data is not likely to be all that useful, because in heterogeneous systems you can no longer figure out which device is really driving the results (e.g. in a system with a Turing GPU and a Maxwell GPU, the former will be heavily penalized by the latter). And it would avoid having to change the presentation of the data much, since results can just be collected under new "2/4/...x GPU Name" devices, similar to the CPUs.

As for how a user could pick that: One idea would be to have the benchmark tool detect if multiple devices with the same name exist and if so, present them as a combined option (while keeping to the radio button selection):
Lets say Cycles reports these devices (so the machine has 3 GPUs + 1 CPU):

RTX 2080Ti (CUDA)
RTX 2080Ti (OptiX)
RTX 2080Ti (CUDA)
RTX 2080Ti (OptiX)
Quadro P6000 (CUDA)
CPU

The benchmark would then present them like this:

RTX 2080Ti (CUDA)           <= This will run the benchmark only on the first RTX 2080Ti
RTX 2080Ti (OptiX)
2x RTX 2080Ti (CUDA)        <= This will run the benchmark on both RTX 2080Ti
2x RTX 2080Ti (OptiX)
Quadro P6000 (CUDA)
CPU

With such a machine set up, right now you could select which RTX 2080Ti to run on, but they should return the same result really (assuming the user put them in correctly), so it would be more useful to replace that with an option to run on both.

Restricting to GPUs of the same kind makes for cleaner benchmark results I think. Otherwise the data is not likely to be all that useful, because in heterogeneous systems you can no longer figure out which device is really driving the results (e.g. in a system with a Turing GPU and a Maxwell GPU, the former will be heavily penalized by the latter). And it would avoid having to change the presentation of the data much, since results can just be collected under new "2/4/...x GPU Name" devices, similar to the CPUs. As for how a user could pick that: One idea would be to have the benchmark tool detect if multiple devices with the same name exist and if so, present them as a combined option (while keeping to the radio button selection): Lets say Cycles reports these devices (so the machine has 3 GPUs + 1 CPU): ``` RTX 2080Ti (CUDA) RTX 2080Ti (OptiX) RTX 2080Ti (CUDA) RTX 2080Ti (OptiX) Quadro P6000 (CUDA) CPU ``` The benchmark would then present them like this: ``` RTX 2080Ti (CUDA) <= This will run the benchmark only on the first RTX 2080Ti RTX 2080Ti (OptiX) 2x RTX 2080Ti (CUDA) <= This will run the benchmark on both RTX 2080Ti 2x RTX 2080Ti (OptiX) Quadro P6000 (CUDA) CPU ``` With such a machine set up, right now you could select which RTX 2080Ti to run on, but they should return the same result really (assuming the user put them in correctly), so it would be more useful to replace that with an option to run on both.

Aaron Carlisle commented

2020-02-17 20:37:53 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'