Category: Collaborative benchmark development

History in the making

on July 13, 2012

We are quickly approaching the debut of HDXPRT 2012. It will be the second version of HDXPRT developed under the benchmark development community paradigm. This milestone provides a nice opportunity to look back at what has happened over the nearly two years since we started creating community-based benchmarks.

The most obvious accomplishment is the development of HDXPRT 2011 and HDXPRT 2012. HDXPRT 2011 has been used around the world for evaluating the performance of computers using applications doing activities that consumers do to create and consume content. We are hopeful that HDXPRT 2012 will be even more widely used.

We also announced earlier this year a new benchmark, TouchXPRT. This benchmark will provide a way to evaluate the performance of the emerging touch-based devices, including tablets. TouchXPRT will debut later this year, initially on Windows 8 Metro.

We have been working hard to get the word out about the benchmarks. We’ve been writing this weekly blog, conducting Webinars, and generally talking with folks in the computer industry. We’ve visited with members of the community around the world at trade shows like CES in Las Vegas and Computex in Taipei. We also spent time with members of the press and computer hardware and software developers. Over the coming months, we are planning to revamp the Web site, add video content, and generally find ways to better engage with and extend the development community.

Less obvious, but equally important to me, has been the development of the development community itself. Developing benchmarks has not been done this way before. We are doing what we can to make the process open to the community, including releasing the benchmark source code. We are optimistic that this method will grow and be a real asset for the industry.

As we look at the growing family of benchmarks under the benchmark XPRT umbrella, the question is always what is next? How can we improve the products and the community? What performance areas do we need to look at in the future? Battery life? Macs? Phones?

Thanks so much for joining us on this journey. The members of this community are what make it work. We look forward to continuing the journey with you!

Bill

Comment on this post in the forums

Posted in Application-based benchmarks, Collaborative benchmark development, HDXPRT, HDXPRT 2012, History of benchmarking, TouchXPRT |

Counting down

By Eric Hale

on July 6, 2012

We’ve been hard at work since the end of the beta period, driving toward the release of HDXPRT 2012. Things are looking good. The RTM is coming soon, so we thought we’d share the next few milestones with you.

RTM candidate: 7/13/12. At this point, we stop development. There are no feature changes after this point.
HDXPRT 2012 launch: 7/27/12. Having tested the RTM and manufactured the DVDs, we mail the benchmark to the community members. This is when the press release goes out. It’s also when we publish the HDXPRT 2012 white paper. Unlike the design document, this paper will explain HDXPRT 2012 to the general public.
Webinar: 8/3/28 – We talk about HDXPRT 2012 and take your questions.
First scaling paper: 8/31/12. As with HDXPRT 2011, we will be publishing a set of scaling studies. The first will test several modern desktop processors, looking at factors such as varying the amount of RAM, comparing HDDs to SSDs, and comparing internal graphics to a couple of popular graphics cards.
Second scaling paper: 9/28/12. In this paper, we expand on the testing we did in the first scaling paper.

We will release an update of HDXPRT 2012 that supports Windows 8 and includes bug fixes within a month of Windows 8 launch.

We’re very excited about the release of HDXPRT 2012, and look forward to seeing what you do with it!

Eric

Comment on this post in the forums

Posted in Application-based benchmarks, Collaborative benchmark development, HDXPRT, HDXPRT 2012, HDXPRT 2012 beta, HDXPRT development process, HDXPRT release cycle |

What to do, what to do

By Mark Van Name

on May 11, 2011

When you set out to build an application-based benchmark like HDXPRT, you face many choices, but two are particularly important: what applications do you run, and what functions do you perform in each application?

With HDXPRT the answers were straightforward, as they should be.

The applications we chose reflected a blend of market leaders, those providing emerging but important features, and the input from our community members.

The functions we perform in each application are ones that are representative of common uses of those programs—and that reflect the input of the community.

What’s so important here is the last clause of each of those paragraphs: your input defines this benchmark.

As we finish off HDXPRT 2011 and then move to the 2012 version, we’ll begin the development cycle anew. When we do, if you want to make sure we choose the applications and functions that matter most to you, then participate, tell us what you want, let us hear your voice. We will respond to all input, so though we can’t guarantee to accept all direction—after all, goals and desires sometimes conflict—we can guarantee that you will hear back from us and that we will explain the rationale for our decisions.

Mark Van Name

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT 2012, HDXPRT development process |

An example of the community in action

By Mark Van Name

on April 27, 2011

Last week, I hosted a Webinar on HDXPRT. We’ll make a recording of it available on the site fairly soon. Multiple members attended. As I was going through the slides and discussing various aspects of the benchmark, a member asked about installing the benchmark from a USB key or a server. My response was the simple truth: we hadn’t considered that approach. As I then elaborated, we clearly should have thought about it, because those capabilities would be useful in just about every production lab out there, including ours here at PT. I concluded by saying that we’d look into it.

I’m not naming the member simply because with big companies I’m never sure if doing that will be good or will cause someone trouble, and I don’t want to cause hassle for anyone. He should, though, feel free to step forward and claim the well-deserved credit for the suggestion.

Less than a week after the Webinar, I’m happy to be able to report that the team has done more than look into these capabilities; it’s implemented them! So, the next Beta release, Beta 2, which we’ll be releasing any time now (maybe even before we post this blog entry), lets you install the benchmark from a network share or a USB key.

I know this is a relatively small thing, but I think it bears reporting because it is exactly the way the community should work. A member brought the benefits of his experience to bear in a great bit of feedback, and now the benchmark is better for it—and so are all of us who use it.

Keep the good ideas coming!

Mark Van Name

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT capabilities, HDXPRT Development Community benefits, HDXPRT development process, HDXPRT Webinar | Tagged Beta, Community |

Our community’s goal

By Bill Catchings

on April 13, 2011

Computer system performance evaluation has a long and complex history. Many of the earliest tests were simple, short code snippets, such as Whetstone, that did little more than give an indication of how fast a particular computer subsystem was able to operate. Unfortunately, such simple benchmarks quickly lost their value, in part because they were very crude measures, and in part because software tools on the things they were measuring could easily optimize for them. In some cases, a compiler could even recognize a test and “optimize” the code by simply producing the final result!

Over time, though, benchmarks have become more complex and more relevant. Whole organizations exist and have existed to build benchmarks. Notable ones include the Ziff-Davis Benchmark Operation (ZDBOp), which the Ziff-Davis computer magazines funded in the 1990s and which Mark and I ran; the Standard Performance Evaluation Corporation (SPEC), which its member companies fund and of which PT is a member; and the Business Applications Performance Corporation (BAPCo), which its member companies fund. Each of these organizations has developed widely used products, such as Winstone (ZDBOp), SPEC CPU (SPEC), and SYSmark (BAPCo). Each organization has also always faced challenges. In the case of ZDBOp, for example, Ziff Davis could no longer support the costs of developing its benchmarks, so they discontinued the group. SPEC continues to develop good benchmarks, but its process can sometimes yield years between versions.

The goal with HDXPRT and the HDXPRT Development Community (HDC) is to explore a new way to develop benchmarks. By utilizing the expertise and experience of a community of interested people, we hope to be able develop benchmarks in an open and collaborative environment while keeping them timely.

HDXPRT 2011 is the first test of this approach. We believe that it and subsequent versions of it, as well as other benchmarks, will give the industry a new model for creating world-class performance measurement tools.

If you’re not a member of the HDC, please consider joining us and helping define the future of performance evaluation.

Bill

Comment on this post in the forums

Posted in Collaborative benchmark development, Future of performance evaluation, HDXPRT development process, Performance benchmarking | Tagged Community, HDXPRT |

Category: Collaborative benchmark development

History in the making

Counting down

What to do, what to do

An example of the community in action

Our community’s goal

Check out the other XPRTs: