Removing Results
I’ve talked about Geekbench on YouTube before. Most of the videos are fun and I enjoy watching them. It’s great to see people getting excited about benchmarks in general and Geekbench in particular.
When a video claims Geekbench is biased, though, the fun goes out the window:
While it’s true I’ve removed results from the Geekbench Result Browser, I’ve never removed results to favor one platform over another. The only results I’ve removed have been results forged by people trying to cheat or results from virtual machines.
Why I’m removing cheaters is pretty obvious, but why am I also removing virtual machine results? Well, virtual machines have problems emulating certain bits of hardware that make it hard, if not impossible, to implement accurate high-resolution timers. For example, if a benchmark takes a second to run, it might look like it only took half a second to run on a virtual machine, which makes the virtual machine look twice as fast.
Right now this isn’t a huge issue. I do a bit of maintenance and keep virtual machine scores out of the top Geekbench scores list. People who use Geekbench competitively (like the folks involved in the Forum Wars) know that Geekbench scores from virtual machines aren’t always accurate.
That said, it’s still an issue, and I’m investigating ways of detecting and correcting inaccurate timers on virtual machines. In the meantime if you’re interested in running benchmarks on virtual machines, or in understanding the issues behind timers on virtual machines, VMWare has a great white paper that explains the issue and how to work around it under VMWare.
Getting back to the original issue, I was really excited when a Windows PC snagged the top spot on the Geekbench Result Browser after months of Mac Pro domination. After all, part of the fun of benchmarks is the competition, and where’s the competition when the the same computer dominates the results?