Category: Collaborative benchmark development

Look how we’ve grown!

on October 10, 2013

We talk a lot about the community model, but perhaps not enough about the community itself. I was looking at our growth over the past few months, and struck by how far we have come this year. We don’t reveal our members’ personal information, but I would like to share some statistics with you.

The community has grown by over 40 percent in 2013. Members are affiliated with 49 organizations, representing chip makers, OEMs, software vendors, and the press. Some members work outside technology, in fields such as engineering, finance, and communications.

Back in April, we blogged about setting up a second WebXPRT host to help with download speeds in China, so it’s no surprise that the community has also spread across the world. Because we don’t require detailed information from community members, I don’t know how many countries are represented. However, the members with whom I’ve had personal contact come from at least four continents.

Thanks to all our community members for making it a great year so far. If you’ve not joined yet, now is a great time. Remember that members see the previews of the benchmarks, and the 2014 versions are coming.

Here’s looking forward to greater growth in the future.

Eric

Comment on this post in the forums

Staying out in the open

By Eric Hale

on October 3, 2013

Back in July, Anandtech publicized some research about possible benchmark optimizations in the Galaxy S4. Yesterday, Anandtech published a much more comprehensive article, “The State of Cheating in Android Benchmarks.” It’s well worth the read.

Anandtech doesn’t accuse any of the benchmarks of being biased—it’s the OEMS who are supposedly doing the optimizations. I will note that none of the XPRT benchmarks are among the whitelisted CPU tests. That being said, I imagine that everyone in the benchmark game is concerned about any implication that their benchmark could be biased.

When I was a kid, my parents taught me that it’s a lot harder to cheat in the open. This is one of the reasons we believe so strongly in the community model for software development. The source code is available to anyone who joins the community. It’s impossible to hide any biases. At the same time, it allows us to control derivative works. That’s necessary to avoid biased versions of the benchmarks being published. We think the community model strikes the right balance.

However, any time there is a system, someone will try to game it. We’ll always be on the lookout for optimizations that happen outside the benchmarks.

Eric

Comment on this post in the forums

Posted in Android, Benchmark metrics, Benchmarks in general, BenchmarkXPRT development community, Collaborative benchmark development, MobileXPRT |

There’s always something new to measure

By Eric Hale

on July 25, 2013

Scientific Reports published an article this month showing that electrically charged insects, such as honey bees, can cause some spider webs to deform. The effect, as pictures in the report show, is that the webs can literally reach out a short distance to snag the passing insect. It’s a cool optimization for the web, and one no one had measured before.

Like the scientists in the report, we are always looking for interesting things to measure. Even as we head toward the public release of MobileXPRT, we’re already thinking ahead to the next versions of all the XPRT benchmarks. This week, I’d like to share a few of the things we have been thinking about, based on conversations with folks (both within and outside of the Development Community) for WebXPRT.

Enhancing the benchmark UI to provide better feedback about the test progress
Extending coverage to other Web technologies such as Web Workers and CSS 3
Extending the workload categories to include other areas, such as productivity
Enabling WebXPRT to be used as a platform-independent battery life test

There are just some of the ideas we’ve been talking about. Which of these excite you? Even better, do you have any cool ideas of your own? Either way, post to the forums or e-mail BenchmarkXPRT@principledtechnologies.com

Eric

Comment on this post in the forums

Posted in Benchmarking, BenchmarkXPRT, Collaborative benchmark development, WebXPRT |

Back to the source

By Eric Hale

on July 4, 2013

Last week, we released MobileXPRT CP1.1.This week, we released the source code for MobileXPRT CP1.1. You can download it here (login required). The procedure for building it is the same as for the previous CP. As we discussed in Kick the tires. Look under the hood, it’s easy to set up the environment and all the necessary software is free.

We believe that one of the most important things we can do is make the source code available. We believe that increasing the transparency of the benchmarking process and stimulating the participation of industry players and the public in the definition, development, understanding, and auditing of the benchmarks will lead to better benchmarks.

You may be thinking “Then why not open source the benchmark?” The short answer is that we need to make sure that the results from any version of our benchmarks are ones you can trust. You can watch Bill discuss this in BenchmarkXPRT: It’s not a benchmark.

We believe that the community model—which gives you total visibility into the benchmark, allows you to run your own experiments and contribute to future versions of the benchmark, and still protects the integrity of the results—strikes the right balance.

If you’re not a member, please consider joining. It’s easy.

If you are a member, check out the code and tell us how it can be better!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT development community, Collaborative benchmark development, MobileXPRT 2013 source, Video |

Presentation XPRT

By Bill Catchings

on May 9, 2013

A few weeks ago, the Mobile PC Extended Battery Life Working Group (EBL WG) asked me to give a presentation about the BenchmarkXPRT benchmarks during one of their meetings. The group started a little over 10 years ago and is an industry organization focusing on achieving all-day battery life in mobile PC platforms.

There was a catch, of course. The meeting was this week, in Taipei. Luckily, we were able to grant the request without racking up any frequent flier miles—thanks to the wonders of technology, I was able to give the presentation remotely.

Taipei is 12 hours ahead of us here on the East Coast, so I made arrangements to give the presentation last night at 9:30 pm, making it 9:30 am the next day there—the first thing on the agenda for today. (Oh, the vagaries of time zones!)

During the one-hour session we talked about HDXPRT, TouchXPRT, WebXPRT and MobileXPRT (which, for the moment, is how we internally refer to the XPRT formerly known as PhoneXPRT). And of course, we discussed the BenchmarkXPRT Development Community. It went well, and we had some great questions from the group.

It went so well, in fact, that I’d love to do it again. I like talking about the development community and the XPRTs, and would welcome the chance to present to other groups. If you are interested or know an organization that is, you can email us at benchmarkxprtsupport@principledtechnologies.com.

Bill

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, HDXPRT, MobileXPRT, PhoneXPRT, TouchXPRT |

The HDXPRT 2013 RFC is here

By Eric Hale

on February 14, 2013

We released the RFC, or request for comments, for HDXPRT 2013 yesterday. Our major objective with the RFC is to get your feedback. Your feedback played an important part in developing HDXPRT 2012, and we are hoping it plays an even larger role in developing HDXPRT 2013.

The RFC includes our thoughts and ideas for the design of HDXPRT 2013 based on the many conversations we’ve had over the six months since the current version of HDXPRT debuted. Indeed, during the last few weeks, we shared some of the feedback we received during and after the Webinar in January.

At this point, nothing is written in stone. Now is the time to let us know where you agree and where you disagree. For example, the current proposal drops support for Windows 7. Do you have an opinion about this? Let us know.

The RFC is available for Development Community members at http://www.principledtechnologies.com/hdxprt/forum/hdxprt2013RFC.php. Our goal is to get your feedback by March 6. We’d like as much of the feedback as possible to appear on the forums to help stimulate discussion. However, if you prefer to send in your comments via email, please send them to BenchmarkXPRTsupport@hdxprt.com.

Of course, you can send comments to us any time, and you don’t have to limit yourself to HDXPRT! Do you have thoughts about TouchXPRT or WebXPRT? They are both moving rapidly toward their official releases. Do you have thoughts about other benchmarks we should consider developing? Send those, too!

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT 2013, HDXPRT 2013 RFC, HDXPRT Development Community benefits, HDXPRT development process |

Category: Collaborative benchmark development

Look how we’ve grown!

Staying out in the open

There’s always something new to measure

Back to the source

Presentation XPRT

The HDXPRT 2013 RFC is here

Check out the other XPRTs: