Category: Benchmarking

Staying out in the open

on October 3, 2013

Back in July, Anandtech publicized some research about possible benchmark optimizations in the Galaxy S4. Yesterday, Anandtech published a much more comprehensive article, “The State of Cheating in Android Benchmarks.” It’s well worth the read.

Anandtech doesn’t accuse any of the benchmarks of being biased—it’s the OEMS who are supposedly doing the optimizations. I will note that none of the XPRT benchmarks are among the whitelisted CPU tests. That being said, I imagine that everyone in the benchmark game is concerned about any implication that their benchmark could be biased.

When I was a kid, my parents taught me that it’s a lot harder to cheat in the open. This is one of the reasons we believe so strongly in the community model for software development. The source code is available to anyone who joins the community. It’s impossible to hide any biases. At the same time, it allows us to control derivative works. That’s necessary to avoid biased versions of the benchmarks being published. We think the community model strikes the right balance.

However, any time there is a system, someone will try to game it. We’ll always be on the lookout for optimizations that happen outside the benchmarks.

Eric

Comment on this post in the forums

Posted in Android, Benchmark metrics, Benchmarks in general, BenchmarkXPRT development community, Collaborative benchmark development, MobileXPRT |

There’s always something new to measure

By Eric Hale

on July 25, 2013

Scientific Reports published an article this month showing that electrically charged insects, such as honey bees, can cause some spider webs to deform. The effect, as pictures in the report show, is that the webs can literally reach out a short distance to snag the passing insect. It’s a cool optimization for the web, and one no one had measured before.

Like the scientists in the report, we are always looking for interesting things to measure. Even as we head toward the public release of MobileXPRT, we’re already thinking ahead to the next versions of all the XPRT benchmarks. This week, I’d like to share a few of the things we have been thinking about, based on conversations with folks (both within and outside of the Development Community) for WebXPRT.

Enhancing the benchmark UI to provide better feedback about the test progress
Extending coverage to other Web technologies such as Web Workers and CSS 3
Extending the workload categories to include other areas, such as productivity
Enabling WebXPRT to be used as a platform-independent battery life test

There are just some of the ideas we’ve been talking about. Which of these excite you? Even better, do you have any cool ideas of your own? Either way, post to the forums or e-mail BenchmarkXPRT@principledtechnologies.com

Eric

Comment on this post in the forums

Posted in Benchmarking, BenchmarkXPRT, Collaborative benchmark development, WebXPRT |

Back to the source

By Eric Hale

on July 4, 2013

Last week, we released MobileXPRT CP1.1.This week, we released the source code for MobileXPRT CP1.1. You can download it here (login required). The procedure for building it is the same as for the previous CP. As we discussed in Kick the tires. Look under the hood, it’s easy to set up the environment and all the necessary software is free.

We believe that one of the most important things we can do is make the source code available. We believe that increasing the transparency of the benchmarking process and stimulating the participation of industry players and the public in the definition, development, understanding, and auditing of the benchmarks will lead to better benchmarks.

You may be thinking “Then why not open source the benchmark?” The short answer is that we need to make sure that the results from any version of our benchmarks are ones you can trust. You can watch Bill discuss this in BenchmarkXPRT: It’s not a benchmark.

We believe that the community model—which gives you total visibility into the benchmark, allows you to run your own experiments and contribute to future versions of the benchmark, and still protects the integrity of the results—strikes the right balance.

If you’re not a member, please consider joining. It’s easy.

If you are a member, check out the code and tell us how it can be better!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT development community, Collaborative benchmark development, MobileXPRT 2013 source, Video |

The truth hurts

By Eric Hale

on June 20, 2013

The truth hurts sometimes. Bill loves to mention how long he, Mark, and I have been involved in benchmark software. I am, of course, grateful to him for reminding me of just how old we all are.

Another truth that hurts is that sometimes things don’t happen when you want them to. Case in point: The preview of Windows 8.1 is supposed to be available on June 26. We released TouchXPRT, which is based on Windows 8, to the public in February.

The problem of timing is not limited to benchmarks. All software has timing issues. While the advance descriptions of the changes in Windows 8.1 don’t mention anything that should break the benchmark, we need to see what, if any, effect, the new version of the OS may have on the benchmark and its results.

As soon as the preview is available, we’ll start testing with it. As soon as we know whether TouchXPRT runs on Windows 8.1 and gives results comparable to Windows 8, we will let you know. We would appreciate help from any of you in trying out TouchXPRT with Windows 8.1. Let us know what you find out!

Eric

Comment on this post in the forums

Posted in Benchmarking, TouchXPRT, Windows 8.1 |

Chaos and opportunity

By Eric Hale

on June 13, 2013

With both E3 and Apple’s WWDC happening this week, there’s been a lot of news. There’s also been a lot of hyperbolic commentary. I am not about to get into the arguments about the PS4 vs. the Xbox One or iOS 7 vs. Android.

It was Tim Cook’s presentation at WWDC that really got my attention. It’s unusual in an executive presentation to focus so much attention on a particular competitor, but Android was clearly on his mind. At one point, he focused harsh attention on fragmentation in the Android market, calling it “terrible” for developers. You can see the video here, at about 74 minutes.

As we saw in the 90s, chaos can breed innovation. At that time, the paradigm was that Macs always worked, but if you wanted the most advanced hardware, you should get a PC. I remember the editors at MacWorld, who deeply, truly loved the Mac, lusting over the (by the standards of the time) small, light, cheap notebooks PC users could get.

That being said, we understand the challenges of developing in the Android market. As I said in It’s finally here!, the Android ecosystem is sufficiently diverse that we know the benchmark will encounter configurations we’ve not seen before. If you have any problems with the MobileXPRT CP, please let us know at benchmarkxprtsupport@principledtechnologies.com. We want the benchmark to be the best it can be.

Eric

Comment on this post in the forums

Posted in Android, BenchmarkXPRT development community, History of benchmarking, History of computers, MobileXPRT |