Basemark Releases VRScore, a VR & VR Headset Benchmark Suite for Windows

Back in 2015 as the development of the first generation of modern VR headsets was coming to a close, benchmark developer Basemark announced that they would be applying their talents to the field of VR. The benchmark, now named VRScore, was to be developed in conjunction with Crytek and would serve as a multi-faceted VR test suite for VR headsets and computers, covering everything from rendering performance to display latency. At the time Basemark was expecting to launch the benchmark in mid-to-late 2016, and while development has taken a bit longer than expected, they are finally releasing version 1.0 of the benchmark this morning.

The final product – or rather the first iteration thereof – is designed to be a high-end AAA-quality benchmark, an unsurprising choice given the use of CryEngine V and the need for benchmarks to be forward-looking. CryEngine V of course introduces support for VR to CryEngine, but it also adds support for DirectX 12 as well. For VRScore, Basemark has played things a little more conservatively, designing the benchmark and its “Sky Harbor” scene around DX12, but including a DX11 mode as well for pre-Windows 10 OSes and headsets that don’t yet work with DX12 mode (which happens to be everything except the Oculus Rift at this time). VRScore has no specific minimum recommended GPU – and Basemark isn’t looking to test against Oculus/Valve’s GTX 1060/RX 480 class GPU performance recommendation – but to sustain 90fps you’ll generally need a GTX 1080 or faster.

Also notable here is that Basemark is looking to support as many PC VR headsets as possible. So this includes not only the Oculus Rift and HTC Vive SDKs, but also the OpenVR and OSVR SDKs. This is an important distinction not only because of the wider compatibility afforded by supporting more SDKs, but because it underscores just how important the SDK is in VR performance. The VR headset SDKs dictate the resolution used – following the best practices for each headset – along of course with controlling how synchronization and features like timewarp/spacewarp work. Consequently at this stage of development, benchmarking an active VR headset is as much an SDK benchmark as it is a GPU or CPU benchmark.

Overall, VRScore is broken down into 3 different types of tests. A 4K “baseline” test run without the headset that is meant to be a more typical system benchmark, a second headset-off test run at headset resolutions and configurations, and finally a headset-on test which runs as a proper VR workload. In the case of the latter two tests, this is particularly interesting as it allows Basemark to actually show the performance cost for VR – how much performance is lost from VR SDK features such as lens distortion, 3D audio, and various synchronization steps. As it turns out, the performance hit is not insignificant.

Gallery: VRScore Results: RX 480, GTX 1060, GTX 1080

Starting things off with AMD’s Radeon RX 480 and Oculus Rift attached, the average framerate with the full test drops 14% when the Rift is enabled. Unsurprisingly, even without the headset the RX 480 was already averaging less than 90fps, but the additional load brings the average framerate down to the mid-60s. Though all-told, the RX 480 doesn’t fare too poorly here; the downscaled resolution subtest, which tests a scene at 80% of the VR SDK’s recommended resolution (~ 2131×1268 for a Rift, almost precisely its native resolution) has the card averaging 89fps with an HMD. This goes to show just how expensive supersampling is, as although these relatively low DPI pentile screens greatly benefit from it, the performance cost is significant

Meanwhile in the NVIDIA camp, the story with the GTX 1060 is pretty much the same, to the point that the RX 480 and GTX 1060 are within a couple of FPS of each other. The GTX 1060 ends up paying a 17% performance penalty here with the Rift enabled, and similarly quickly perks up once Basemark pulls back on the resolution some. As for the GTX 1080, its high performance means the card has little trouble averaging 89fps with the benchmark’s default test, hitting the 90Hz refresh rate cap much of the time. Put another way, with a 131fps frame rate with the headset turned off, the GTX 1080 has more than enough performance to pay the price of VR overhead.

Moving on, as I mentioned before the other major test in VRScore is the VRTrek Suite, Basemark’s VR headset evaluation tool. Whereas VRScore measures the headset itself, specifically the application-to-photon latency, dropped frames, and duplicated frames.

To measure this, the VRTrek Suite uses the VRTrek tool, a curious device composed of a pair of phototransistors and plugs into a microphone jack. Intended to simulate the human eye, the VRTrek device is what gives the software feedback on the VR headset’s performance. Phototransistors of course aren’t cameras, so they can’t see/report a full image, but they are sensitive enough to see the cues Basemark puts in the rendered image for headset testing.

Measuring input latency and dropped frames goes as far back as the original proposal for VRScore, but it’s interesting that Basemark has opted to follow-through with it. Competitor Futuremark developed a similar test during development of their VRMark benchmark, but they ultimately scaled it back to just industry use, saying that “measuring the latency of popular headsets does not provide meaningful insight into the actual VR experience” and instead focusing on subjective/experiential testing, especially as modern VR SDKs employ a number of tricks to reduce the perceived latency. As a result, the VRTrek is a fairly unique device since Basemark intends to make it accessible (though I suspect not cheap) outside of the usual industry circles.

I haven’t had the chance to use the VRTrek on headsets from multiple vendors yet, but in testing it against the Oculus Rift, it does what it sets out to do. At the moment I’m not sure how valuable that’s going to be, but as Microsoft is trying to encourage cheap(er) VR with their $300 headset initiative, this will likely prove useful in quantifying just how low the latency is of these forthcoming headsets.

Wrapping things up, while today’s announcement marks the formal launch of VRScore, in practice Basemark is dividing up the launch into two parts. Launching immediately are the full-featured corporate and media versions. Launching a bit farther down the line will be the consumer versions, both free and professional. As with some of Basemark’s other benchmarks, they are offering a free version of the benchmark with a single test/report (the system score) and the VR experience mode, meanwhile reporting of the scores for the sub-feature tests and custom configurations will require a paid version.