Category: Benchmarking

Presentation XPRT

on May 9, 2013

A few weeks ago, the Mobile PC Extended Battery Life Working Group (EBL WG) asked me to give a presentation about the BenchmarkXPRT benchmarks during one of their meetings. The group started a little over 10 years ago and is an industry organization focusing on achieving all-day battery life in mobile PC platforms.

There was a catch, of course. The meeting was this week, in Taipei. Luckily, we were able to grant the request without racking up any frequent flier miles—thanks to the wonders of technology, I was able to give the presentation remotely.

Taipei is 12 hours ahead of us here on the East Coast, so I made arrangements to give the presentation last night at 9:30 pm, making it 9:30 am the next day there—the first thing on the agenda for today. (Oh, the vagaries of time zones!)

During the one-hour session we talked about HDXPRT, TouchXPRT, WebXPRT and MobileXPRT (which, for the moment, is how we internally refer to the XPRT formerly known as PhoneXPRT). And of course, we discussed the BenchmarkXPRT Development Community. It went well, and we had some great questions from the group.

It went so well, in fact, that I’d love to do it again. I like talking about the development community and the XPRTs, and would welcome the chance to present to other groups. If you are interested or know an organization that is, you can email us at benchmarkxprtsupport@principledtechnologies.com.

Bill

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, HDXPRT, MobileXPRT, PhoneXPRT, TouchXPRT |

A happy coincidence

By Bill Catchings

on May 2, 2013

I love new gadgets and even the promise of new ones. Samsung just announced the specs for their upcoming Galaxy Tab 3. Initial reactions to the specs have been somewhat muted to say the least. Basically, they’ve looked at this as only an incremental improvement to the current model. The early rumors of a larger screen and bigger improvements, which turned out to be false, surely contributed to the disappointed reactions.

That being said, some sites claim that the performance of the Galaxy Tab 3 is much higher than the Galaxy Tab 2, particularly regarding graphics. We look forward to verifying these claims ourselves.

Coincidentally, this week we have been playing with an early version of PhoneXPRT (or whatever we end up calling it). So far, things are looking good. We ran it on several devices, including a Samsung Galaxy Tab 2. Like all the XPRT benchmarks, it uses real- world scenarios, which we think result in more useful and accurate results. We’ll talk more about the scenarios in the next few weeks.

It’s a very exciting time in the Android phone and tablet market. I can’t wait to try out subsequent versions of the new benchmark on the latest and greatest Android devices!

Bill

Comment on this post in the forums

Posted in MobileXPRT, Performance testing on tablets, PhoneXPRT, Touch-based devices |

Lies, damned lies, and statistics

By Eric Hale

on April 11, 2013

No one knows who first said “lies, damned lies, and statistics,” but it’s easy to understand why they said it. It’s no surprise that the bestselling statistics book in history is titled How to Lie with Statistics. While the title is facetious, it is certainly true that statistics can be confusing—consider the word “average,” which can refer to the mean, median, or mode. “Mean average,” in turn, can refer to the arithmetic mean, the geometric mean, or the harmonic mean. It’s enough to make a non-statistician’s head spin.

In fact, a number of people have been confused by the confidence interval WebXPRT reports. We believe that the best way to stand behind your results is to be completely open about how you crunch the numbers. To this end, we released the white paper WebXPRT 2013 results calculation and confidence interval this past Monday.

This white paper, which does not require a background in mathematics, explains what the WebXPRT confidence interval is and how it differs from the benchmark variability we sometimes talk about. The paper also gives an overview of the statistical and mathematical techniques WebXPRT uses to translate the raw timing numbers into results.

Because sometimes the devil is in the details, we wanted to augment our overview by showing exactly how WebXPRT calculates results. The white paper is accompanied by a spreadsheet that reproduces the calculations WebXPRT uses. If you are mathematically inclined and would like to suggest improvements to the process, by all means let us know!

Eric

Comment on this post in the forums

Posted in Benchmark metrics, WebXPRT, WebXPRT 2013 results, White papers |

The HDXPRT 2013 RFC is here

By Eric Hale

on February 14, 2013

We released the RFC, or request for comments, for HDXPRT 2013 yesterday. Our major objective with the RFC is to get your feedback. Your feedback played an important part in developing HDXPRT 2012, and we are hoping it plays an even larger role in developing HDXPRT 2013.

The RFC includes our thoughts and ideas for the design of HDXPRT 2013 based on the many conversations we’ve had over the six months since the current version of HDXPRT debuted. Indeed, during the last few weeks, we shared some of the feedback we received during and after the Webinar in January.

At this point, nothing is written in stone. Now is the time to let us know where you agree and where you disagree. For example, the current proposal drops support for Windows 7. Do you have an opinion about this? Let us know.

The RFC is available for Development Community members at http://www.principledtechnologies.com/hdxprt/forum/hdxprt2013RFC.php. Our goal is to get your feedback by March 6. We’d like as much of the feedback as possible to appear on the forums to help stimulate discussion. However, if you prefer to send in your comments via email, please send them to BenchmarkXPRTsupport@hdxprt.com.

Of course, you can send comments to us any time, and you don’t have to limit yourself to HDXPRT! Do you have thoughts about TouchXPRT or WebXPRT? They are both moving rapidly toward their official releases. Do you have thoughts about other benchmarks we should consider developing? Send those, too!

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT 2013, HDXPRT 2013 RFC, HDXPRT Development Community benefits, HDXPRT development process |

Straight from the source

By Eric Hale

on February 7, 2013

One of the pillars of our community model of benchmark development is making the source available. As we’ve said many times, we believe that doing so leads to better benchmarks.

Today we released the source for HDXPRT 2012. As with previous versions of HDXPRT, the source is available only to community members, not to the general public. We apologize that it has taken so long. HDXPRT is complicated to build, and we wanted to have a simpler and more robust build process before we made the source available.

The source allows you to examine how HDXPRT is implemented and to try some experiments of your own. Because of the size of HDXPRT 2012, the source package does not include the applications or the data files for the workloads. By including only the benchmark source code and associated files, we could keep the package small enough to download. If you want to try some changes for experiments and test them, all you need to do is install HDXPRT 2012 from the distribution DVDs. The compilation instructions will tell you how to copy your modified executables over the shipping versions.

Community members can get instructions on how to download the source code here (registration required).

If you create something interesting while you’re experimenting, let us know! We’d love to have the community consider it for HDXPRT 2013.

Speaking of the community, we’ve sent T-shirts to all community members who’ve supplied their up-to-date mailing address. If you’re a community member who wants a shirt but hasn’t yet let us know, please e-mail benchmarkxprtsupport@principledtechnologies.com with your mailing address by February 15th.

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT, HDXPRT Development Community benefits, HDXPRT source code |

Keep them coming!

By Eric Hale

on January 31, 2013

Questions and comments have continued to come in since the Webinar last week. Here are a few of them:

How long are results valid? For a reviewer like us, we need to know that we can reuse results for a reasonable length of time. There is a tension between keeping results stable and keeping the benchmark current enough for the results to be relevant. Historically, HDXPRT allowed at least a year between releases. Based on the feedback we’ve received, a year seems like a reasonable length of time.
Is HDXPRT command line operable? (asked by a community member with a scripted suite of tests) HDXPRT 2012 is not, but we will consider adding a command line interface for HDXPRT 2013. While most casual users don’t need a command line interface, it could be very valuable to those of us using HDXPRT in labs.
I would be hesitant to overemphasize the running time of HDXPRT. The more applications it runs, the more it can differentiate things and the more interesting it is to those of us who run it at a professional level. If I could say “This gives a complete overview of the performance of this system,” that would actually save time. This comment was a surprise, given the amount of feedback we received saying that HDXPRT was too large. However, this gets to the heart of why we all need to be careful as we consider which applications to include in HDXPRT 2013.

If you had to miss the Webinar, it’s available at the BenchmarkXPRT 2013 Webinars page.

We’re planning to release the HDXPRT 2013 RFC next week. We’re looking forward to your comments.

Eric

Comment on this post in the forums

Posted in Benchmark metrics, Benchmarking, HDXPRT, HDXPRT workloads |

Category: Benchmarking

Presentation XPRT

A happy coincidence

Lies, damned lies, and statistics

The HDXPRT 2013 RFC is here

Straight from the source

Keep them coming!

Check out the other XPRTs: