Author Archives: Eric Hale

Sprucing things up

on October 17, 2013

A major goal of the BenchmarkXPRT benchmarks is to be useful to the people who buy consumer devices, most of whom do not have experience running benchmarks. As these people have started discovering the benchmarks, we’ve been learning a lot about how to make the benchmarks clearer and easier to use.

We’re working on a lot of things: making the data clearer and easier to understand, giving better feedback about what the benchmark is doing, making the controls clearer and easier to find, making the documentation more accessible, and more. We also need to make the UIs of the benchmarks more unified, so that the benchmarks appear to be the members of the same family.

Because we take this effort very seriously, we are involving the PT design team in a more integral way. These are the same people who spearheaded the Web site redesign and produced the BenchmarkXPRT videos. (See their most recent video here.)

We’ll keep you informed about the design process as we go. If you have complaints or suggestions, it’s not too late. Please post to the forums or send an email to benchmarkxprtsupport@principledtechnologies.com.

Comment on this post in the forums

Look how we’ve grown!

By Eric Hale

on October 10, 2013

We talk a lot about the community model, but perhaps not enough about the community itself. I was looking at our growth over the past few months, and struck by how far we have come this year. We don’t reveal our members’ personal information, but I would like to share some statistics with you.

The community has grown by over 40 percent in 2013. Members are affiliated with 49 organizations, representing chip makers, OEMs, software vendors, and the press. Some members work outside technology, in fields such as engineering, finance, and communications.

Back in April, we blogged about setting up a second WebXPRT host to help with download speeds in China, so it’s no surprise that the community has also spread across the world. Because we don’t require detailed information from community members, I don’t know how many countries are represented. However, the members with whom I’ve had personal contact come from at least four continents.

Thanks to all our community members for making it a great year so far. If you’ve not joined yet, now is a great time. Remember that members see the previews of the benchmarks, and the 2014 versions are coming.

Here’s looking forward to greater growth in the future.

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, WebXPRT |

Staying out in the open

By Eric Hale

on October 3, 2013

Back in July, Anandtech publicized some research about possible benchmark optimizations in the Galaxy S4. Yesterday, Anandtech published a much more comprehensive article, “The State of Cheating in Android Benchmarks.” It’s well worth the read.

Anandtech doesn’t accuse any of the benchmarks of being biased—it’s the OEMS who are supposedly doing the optimizations. I will note that none of the XPRT benchmarks are among the whitelisted CPU tests. That being said, I imagine that everyone in the benchmark game is concerned about any implication that their benchmark could be biased.

When I was a kid, my parents taught me that it’s a lot harder to cheat in the open. This is one of the reasons we believe so strongly in the community model for software development. The source code is available to anyone who joins the community. It’s impossible to hide any biases. At the same time, it allows us to control derivative works. That’s necessary to avoid biased versions of the benchmarks being published. We think the community model strikes the right balance.

However, any time there is a system, someone will try to game it. We’ll always be on the lookout for optimizations that happen outside the benchmarks.

Eric

Comment on this post in the forums

Posted in Android, Benchmark metrics, Benchmarks in general, BenchmarkXPRT development community, Collaborative benchmark development, MobileXPRT |

Interesting questions

By Eric Hale

on September 26, 2013

We’ve had a couple of interesting questions about WebXPRT this week.

The first question was about the Face detect test in WebXPRT. One person, having noticed that changing the version of Firefox affected the WebXPRT score on a particular device, asked whether the test used the JavaScript Canvas element. The answer is yes, the Face detection test does use the Canvas element. It is based on the JavaScript library by Dr. Liu Liu.

As we have discussed in the past, the software stack on a device affects the benchmark scores. WebXPRT is a HTML5 benchmark and uses elements in the HTML5 specification, such as Canvas. Browsers implement HTML in their JavaScript engines, whose performance depends on the OS and the underlying platform. So, WebXPRT scores are influenced by the browser and OS, as well as the platform.

The second question was whether it is possible to run WebXPRT without an Internet connection. Generally speaking, the answer to that is no. WebXPRT is a hosted application, and to run the official version, you must be able to connect to the WebXPRT servers.

However, community members can download the WebXPRT source and configure local servers that will run WebXPRT, if they desire. Note: As we discussed in Sources, any published results must be from the version hosted at webxprt.com.

Thanks for the questions and keep experimenting!

Eric

Comment on this post in the forums

Posted in HTML5, WebXPRT |

Sources

By Eric Hale

on September 19, 2013

If you’ve checked out the MobileXPRT and WebXPRT pages recently, you’ve probably noticed that the number of results has started to grow. The results are coming from three sources:

Internal testing at PT.
Results submitted by the public.
Results published on the Web. We link back to the source from these results. Results published on-line include results in reports PT publishes for clients and reviews by other parties.

While we are excited about the growing number of results, we do sanity check them. We compare the results with other runs for the same device when available, or with similar devices if not.

The source code for the benchmarks is available, and we encourage experimentation. However, it should go without saying that valid runs must come from the builds of the benchmarks the development community has published. We can’t compare two results generated by different builds.

That being said, if you change the code and get an interesting result, by all means do contact us. You may have discovered something that we’ll want to include in a future version

Keep the results coming and keep experimenting!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, MobileXPRT, TouchXPRT, WebXPRT |

Improvements

By Eric Hale

on September 12, 2013

We recently made some changes to how MobileXPRT installs. Previously, when you installed MobileXPRT from Google Play, it downloaded the user experience (UX) tests from the Principled Technologies servers. This required you to have the Unknown sources option set, which allows installation of non-market apps, even if you installed MobileXPRT from Google Play.

We have removed this restriction. MobileXPRT now installs the UX tests from Google play, so installation is cleaner. However, the UX tests are not intended to be installed outside of MobileXPRT.

An unrelated fix now makes sure that MobileXPRT cleans out all of its content when you uninstall it. This fix required the location of the results file to move. See Submit your MobileXPRT 2013 results! for details.

If you prefer, you can still install MobileXPRT from the PT site. However, when installing from the PT site, you will still need the Unknown sources option set.

We continue looking for ways to improve the benchmarks. If you have any things you’d like to see us improve, please post to the forums or send an e-mail to benchmarkxprtsupport@principledtechnologies.com

Eric

Comment on this post in the forums

Posted in MobileXPRT |

Author Archives: Eric Hale

Sprucing things up

Look how we’ve grown!

Staying out in the open

Interesting questions

Sources

Improvements

Check out the other XPRTs: