BenchmarkXPRT Blog banner

Category: Future of performance evaluation

Home sweet home

After a long set of flights back from Computex in Taipei, I’m finally home in North Carolina. Unfortunately, I’m still not quite sure what time zone I’m in!

While awake in the middle of the night, I’ve been thinking about some of the things I saw at Computex.  While I was there, it seemed like a jumble of notebooks, power supplies, gaming rigs, motherboards, cases, Hello Kitty accessories, and some things that I still don’t quite know what they were.   Many of the things I saw were not brand new, but it was my first chance to see them up close.  Some of them were of technologies still on the horizon like Intel’s Ultrabook concept and Microsoft’s Windows 8.  I also saw all sorts of combinations of phones, 4G, and other devices.

One thing that stood out to me were the number and variety of tablets.  They were in a variety of sizes (and screen resolutions).  There were quite a few vendors and some were ones I would not have suspected but was pleasantly surprised to encounter, like Viewsonic and Shuttle.  The OS choices included Android, WebOS, and MeeGo.   ASUS had a couple of interesting hybrid approaches such as the Eee Pad Transformer and the Padfone.  The former is a 10.1-inch tablet that plugs into a keyboard.  The Padfone is a smartphone that can plug into the back of a larger (10.1-inch) touch screen to act as a tablet.

All of these tablet choices, as well as the iPad that they all must be compete against, left me wondering how to choose between them.  Some part of the choice comes down to the size and features.  As always, however, performance plays a key role.  My tolerance for waiting on a tablet device is even lower than it is for waiting on my PC.  The problem is how to make valid comparisons across such a wide range of platforms.  I’d love to hear from you what you think about performance testing on tablets.  Is it useful?  What are the best ways to accomplish it?

Finally, thanks to all the folks who came by and visited our suite at Computex.  I enjoyed getting the chance to meet some of the members of the HDXPRT Development Community.  And, hopefully, I convinced more folks to join.

Bill

Comment on this post in the forums

Our community’s goal

Computer system performance evaluation has a long and complex history. Many of the earliest tests were simple, short code snippets, such as Whetstone, that did little more than give an indication of how fast a particular computer subsystem was able to operate. Unfortunately, such simple benchmarks quickly lost their value, in part because they were very crude measures, and in part because software tools on the things they were measuring could easily optimize for them. In some cases, a compiler could even recognize a test and “optimize” the code by simply producing the final result!

Over time, though, benchmarks have become more complex and more relevant. Whole organizations exist and have existed to build benchmarks. Notable ones include the Ziff-Davis Benchmark Operation (ZDBOp), which the Ziff-Davis computer magazines funded in the 1990s and which Mark and I ran; the Standard Performance Evaluation Corporation (SPEC), which its member companies fund and of which PT is a member; and the Business Applications Performance Corporation (BAPCo), which its member companies fund. Each of these organizations has developed widely used products, such as Winstone (ZDBOp), SPEC CPU (SPEC), and SYSmark (BAPCo). Each organization has also always faced challenges. In the case of ZDBOp, for example, Ziff Davis could no longer support the costs of developing its benchmarks, so they discontinued the group. SPEC continues to develop good benchmarks, but its process can sometimes yield years between versions.

The goal with HDXPRT and the HDXPRT Development Community (HDC) is to explore a new way to develop benchmarks. By utilizing the expertise and experience of a community of interested people, we hope to be able develop benchmarks in an open and collaborative environment while keeping them timely.

HDXPRT 2011 is the first test of this approach. We believe that it and subsequent versions of it, as well as other benchmarks, will give the industry a new model for creating world-class performance measurement tools.

If you’re not a member of the HDC, please consider joining us and helping define the future of performance evaluation.

Bill

Comment on this post in the forums

Check out the other XPRTs:

Forgot your password?