Category: Benchmarking

The truth hurts

on June 20, 2013

The truth hurts sometimes. Bill loves to mention how long he, Mark, and I have been involved in benchmark software. I am, of course, grateful to him for reminding me of just how old we all are.

Another truth that hurts is that sometimes things don’t happen when you want them to. Case in point: The preview of Windows 8.1 is supposed to be available on June 26. We released TouchXPRT, which is based on Windows 8, to the public in February.

The problem of timing is not limited to benchmarks. All software has timing issues. While the advance descriptions of the changes in Windows 8.1 don’t mention anything that should break the benchmark, we need to see what, if any, effect, the new version of the OS may have on the benchmark and its results.

As soon as the preview is available, we’ll start testing with it. As soon as we know whether TouchXPRT runs on Windows 8.1 and gives results comparable to Windows 8, we will let you know. We would appreciate help from any of you in trying out TouchXPRT with Windows 8.1. Let us know what you find out!

Eric

Comment on this post in the forums

Posted in Benchmarking, TouchXPRT, Windows 8.1 |

Chaos and opportunity

By Eric Hale

on June 13, 2013

With both E3 and Apple’s WWDC happening this week, there’s been a lot of news. There’s also been a lot of hyperbolic commentary. I am not about to get into the arguments about the PS4 vs. the Xbox One or iOS 7 vs. Android.

It was Tim Cook’s presentation at WWDC that really got my attention. It’s unusual in an executive presentation to focus so much attention on a particular competitor, but Android was clearly on his mind. At one point, he focused harsh attention on fragmentation in the Android market, calling it “terrible” for developers. You can see the video here, at about 74 minutes.

As we saw in the 90s, chaos can breed innovation. At that time, the paradigm was that Macs always worked, but if you wanted the most advanced hardware, you should get a PC. I remember the editors at MacWorld, who deeply, truly loved the Mac, lusting over the (by the standards of the time) small, light, cheap notebooks PC users could get.

That being said, we understand the challenges of developing in the Android market. As I said in It’s finally here!, the Android ecosystem is sufficiently diverse that we know the benchmark will encounter configurations we’ve not seen before. If you have any problems with the MobileXPRT CP, please let us know at benchmarkxprtsupport@principledtechnologies.com. We want the benchmark to be the best it can be.

Eric

Comment on this post in the forums

Posted in Android, BenchmarkXPRT development community, History of benchmarking, History of computers, MobileXPRT |

The MobileXPRT 2013 community preview is available!

By Eric Hale

on May 29, 2013

If you’re a member, you can download it here. If you’re not a member, join and see what the excitement’s about.

Comment on this post in the forums

Posted in Community Preview, MobileXPRT |

Presentation XPRT

By Bill Catchings

on May 9, 2013

A few weeks ago, the Mobile PC Extended Battery Life Working Group (EBL WG) asked me to give a presentation about the BenchmarkXPRT benchmarks during one of their meetings. The group started a little over 10 years ago and is an industry organization focusing on achieving all-day battery life in mobile PC platforms.

There was a catch, of course. The meeting was this week, in Taipei. Luckily, we were able to grant the request without racking up any frequent flier miles—thanks to the wonders of technology, I was able to give the presentation remotely.

Taipei is 12 hours ahead of us here on the East Coast, so I made arrangements to give the presentation last night at 9:30 pm, making it 9:30 am the next day there—the first thing on the agenda for today. (Oh, the vagaries of time zones!)

During the one-hour session we talked about HDXPRT, TouchXPRT, WebXPRT and MobileXPRT (which, for the moment, is how we internally refer to the XPRT formerly known as PhoneXPRT). And of course, we discussed the BenchmarkXPRT Development Community. It went well, and we had some great questions from the group.

It went so well, in fact, that I’d love to do it again. I like talking about the development community and the XPRTs, and would welcome the chance to present to other groups. If you are interested or know an organization that is, you can email us at benchmarkxprtsupport@principledtechnologies.com.

Bill

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, HDXPRT, MobileXPRT, PhoneXPRT, TouchXPRT |

A happy coincidence

By Bill Catchings

on May 2, 2013

I love new gadgets and even the promise of new ones. Samsung just announced the specs for their upcoming Galaxy Tab 3. Initial reactions to the specs have been somewhat muted to say the least. Basically, they’ve looked at this as only an incremental improvement to the current model. The early rumors of a larger screen and bigger improvements, which turned out to be false, surely contributed to the disappointed reactions.

That being said, some sites claim that the performance of the Galaxy Tab 3 is much higher than the Galaxy Tab 2, particularly regarding graphics. We look forward to verifying these claims ourselves.

Coincidentally, this week we have been playing with an early version of PhoneXPRT (or whatever we end up calling it). So far, things are looking good. We ran it on several devices, including a Samsung Galaxy Tab 2. Like all the XPRT benchmarks, it uses real- world scenarios, which we think result in more useful and accurate results. We’ll talk more about the scenarios in the next few weeks.

It’s a very exciting time in the Android phone and tablet market. I can’t wait to try out subsequent versions of the new benchmark on the latest and greatest Android devices!

Bill

Comment on this post in the forums

Posted in MobileXPRT, Performance testing on tablets, PhoneXPRT, Touch-based devices |

Lies, damned lies, and statistics

By Eric Hale

on April 11, 2013

No one knows who first said “lies, damned lies, and statistics,” but it’s easy to understand why they said it. It’s no surprise that the bestselling statistics book in history is titled How to Lie with Statistics. While the title is facetious, it is certainly true that statistics can be confusing—consider the word “average,” which can refer to the mean, median, or mode. “Mean average,” in turn, can refer to the arithmetic mean, the geometric mean, or the harmonic mean. It’s enough to make a non-statistician’s head spin.

In fact, a number of people have been confused by the confidence interval WebXPRT reports. We believe that the best way to stand behind your results is to be completely open about how you crunch the numbers. To this end, we released the white paper WebXPRT 2013 results calculation and confidence interval this past Monday.

This white paper, which does not require a background in mathematics, explains what the WebXPRT confidence interval is and how it differs from the benchmark variability we sometimes talk about. The paper also gives an overview of the statistical and mathematical techniques WebXPRT uses to translate the raw timing numbers into results.

Because sometimes the devil is in the details, we wanted to augment our overview by showing exactly how WebXPRT calculates results. The white paper is accompanied by a spreadsheet that reproduces the calculations WebXPRT uses. If you are mathematically inclined and would like to suggest improvements to the process, by all means let us know!

Eric

Comment on this post in the forums

Posted in Benchmark metrics, WebXPRT, WebXPRT 2013 results, White papers |

Category: Benchmarking

The truth hurts

Chaos and opportunity

The MobileXPRT 2013 community preview is available!

Presentation XPRT

A happy coincidence

Lies, damned lies, and statistics

Check out the other XPRTs: