“How come your benchmark ranks devices differently than [insert other benchmark here]?” It’s a fair question, and the reason is that each benchmark has its own emphasis and tests different things. When you think about it, it would be unusual if all benchmarks did agree.
To illustrate the phenomenon, consider this excerpt from a recent browser shootout in VentureBeat:
- SunSpider: Edge wins!
- Octane: Edge wins!
- Kraken: Chrome wins!
- JetStream: Edge wins!
- Oort Online: Chrome wins!
- Peacekeeper: Firefox wins!
- WebXPRT: Firefox wins!
- HTML5Test: Chrome wins!
While this looks very confusing, the simple explanation is that the different benchmarks are testing different things. To begin with, SunSpider, Octane, JetStream, PeaceKeeper, and Kraken all measure JavaScript performance. Oort Online measures WebGL performance. WebXPRT measures both JavaScript and HTML 5 performance. HTML5Test measures HTML5 compliance.
Even with benchmarks that test the same aspect of browser performance, the tests differ. Kraken and SunSpider both test the speed of JavaScript math, string, and graphics operations in isolation, but run different sets of tests to do so. PeaceKeeper profiles the JavaScript from sites such as YouTube and FaceBook.
WebXPRT, like the other XPRTs, uses scenarios that model the types of work people do with their devices.
It’s no surprise that the order changes depending on which aspect of the Web experience you emphasize, in much the same way that the most fuel-efficient cars might not be the ones with the best acceleration.
This is a bigger topic than we can deal with in a single blog post, and we’ll examine it more in the future.
Eric
