Category: HDXPRT Development Community benefits

Who is on board?

on January 20, 2012

While talking with people at CES about HDXPRT and the upcoming touch benchmark, I encountered the same question a few times—Who are the current members of the Development Community? My answer was something along the lines of “About 10 PC hardware vendors, about the same number of press people, and a few other folks from companies around the world.” I was, however, itching to name the companies because the list is really pretty impressive. We haven’t asked for permission from the Development Community members, though, so I left my answer vague.

Given our goal of expanding the Development Community, I find myself weighing two possible outcomes if we were to make public the names of the companies represented. On the one hand, it could encourage others to join us (“All the other cool kids are doing it, I guess I will too!”). On the other hand, it could discourage others from joining us (“Not sure how my company would feel about this. Should I ask Legal? I’m too busy, never mind.”)

My best plan for now is to email each member individually and ask where he or she stands on company anonymity. And to give all new members the option of keeping their affiliation off the record. Rest assured that we will definitely not reveal this information without your permission.

We’d like to know what you think. Would you have joined the Development Community if doing so required identifying your company and allowing us to share it? Would you now be willing to let us say that someone from your company is a member?

Bill

Comment on this post in the forums

What to do with all the times

By Bill Catchings

on November 17, 2011

HDXPRT, like most other application-based benchmarks, works by timing lots of individual operations. Some other benchmarks just time the entire script. The downside of that approach is that the time includes things that are constant regardless of the speed of the underlying hardware. Some things, like how fast a menu drops down or text scrolls, are tied to the user experience and should not go faster on a faster system. Including those items in the overall time dilutes the importance of the operations that we wait on and are frustrated by, the operations we need to time.

In the case of HDXPRT 2011, we time between 20 and 30 operations. We then roll these up into the times we report as well as the overall score. We do not, however, report the individual times. We expect to include even more timed operations in HDXPRT 2012. As we have been thinking about what the right metrics are, we have started to wonder what to do with all of those times. We could total up the times of similar operations and create additional results. For example, we could total up all the application load times and produce an application-load result. Or, we could total up all the times for an individual application and produce an application result. I can definitely see value in results like those.

Another possibility is to try and look at the general pattern of the results to understand responsiveness. One way would be to collect the times in a histogram, where buckets correspond to ranges of response times for the operations. Such a histogram might give a sense of how responsive a target system feels to an end user. There are certainly other possibilities as well.

If nothing else, I think it makes sense to expose these times in some way. If we make them available, I’m confident that people will find ways to use them. My concern is the danger of burdening a benchmark with too many results. The engineer in me loves all the data possible. The product designer knows that focus is critical. Successful benchmarks have one or maybe two results. How to balance the two?

One wonder of this benchmark development community is the ability to ask you what you think. What would you prefer, simple and clean or lots of numbers? Maybe a combination where we just have the high-level results we have now, but also make other results or times available in an “expert” or an “advanced” mode? What do you think?

Bill

Comment on this post in the forums

Posted in HDXPRT Development Community benefits, HDXPRT metrics, HDXPRT workloads, Let us know your thoughts |

Getting to the source

By Bill Catchings

on September 16, 2011

Many of the earliest benchmarks came in source code form. Dhrystone and many others relied on the compiler for optimization. In fact, some compilers even recognized the code and basically optimized it to a few lines of code that did nothing but return the result! Even some modern benchmarks, such as SPEC CPU and LINPACK, come in source code form.

The source code to application benchmarks, however, has not typically been available. Two of the leading benchmarks of the last twenty years, Winstone and SYSmark, were never available in source code form. The makers of those tools had good reasons for keeping the code private; we know, because led the creation of Winstone. Keeping code private protects your intellectual investment, can make it easier to hit development schedules, and provides many other advantages.

It also, however, can lead some people to criticize that the reason you’re not showing the source code is that it is in some way biased. In benchmarks as in so many areas, transparency is the best way to allay such concerns.

Which leads us to today’s big announcement

We want HDXPRT to be as open as possible, so we’re bucking the normal practice for application-based benchmarks and planning to make the HDXPRT 2011 source code available to the HDXPRT Development Community.

The code will include both the benchmark harness and the scripts that drive the applications. You’ll be able to study everything about the benchmark. You’ll also be able to more easily contribute new code. Which is exactly what we hope you’ll do. We want you not only to be completely comfortable with the benchmark, we want you to contribute to future versions of it.

There will, of course, be some ground rules. We are making the code available only to the HDXPRT Development Community. (If you’re not already a member, joining is cheap and easy: just go here.) Because we want to limit the code to the community, to get access to it, members will have to agree to a license agreement that prevents them from releasing it to the public.

We don’t have an exact schedule in place yet, but over the next week or two, we should have all the necessary things in place to make the source code available.

When you’ve had a chance to look at it, please let us know what improvements you would like to see in HDXPRT 2012. We’ll discuss that version, and how you can help, in the coming weeks.

Bill

Comment on this post in the forums

Posted in Benchmarks in general, HDXPRT, HDXPRT Development Community benefits, HDXPRT source code |

Top 5 reasons for meeting us at Computex in Taipei

By Mark Van Name

on May 18, 2011

As I’ve mentioned before, Bill Catchings from PT will be at the upcoming Computex show in Taipei to debut HDXPRT 2011. At the same time, back home in North Carolina we’ll be mailing copies of the benchmark DVDs to all the members of the HDXPRT Development Community.

If you’re one of the lucky folks who gets to attend Computex, we’d love it if you would come by Bill’s room in the Hyatt (we’ll publicize the room number as soon as we know it), see the benchmark in action, and give us your thoughts about it. I know the show is huge and full of attractions, so I thought I’d give you the top five reasons you ought to make room in your schedule to visit with us.

5. Free snacks! We don’t know what they are yet, or even how we’ll persuade the hotel to let us have them, but we’re committed to providing something to quench your thirst and something to quell your hunger.

4. A break from the crowds. Not only do you get to sit, drink, eat, and see a great new benchmark, you get to do so in the quiet and luxury of a Taipei hotel suite. No more bumping shoulders with fellow show attendees or fighting to get to a place quiet enough that you can talk; in that room, you can relax.

3. You can affect the industry! The support for HDXPRT is growing. More and more organizations are using it. We don’t just want to show it to you; we want you to tell us what you think about it. Your opinions count, and they could help drive the design of the next version of the benchmark, HDXPRT 2012. Yeah, that’s right: the one in development isn’t out, and I’m already talking about the next one. Sue me: I like to live on the edge.

2. You don’t want to make Bill cry. Imagine him, sitting alone in the room, laptop humming, ready to demonstrate this cool new testing tool, and no one to keep him company. His sadness would be so unbearable that I can’t bear to think of what he might do. You can’t let that happen.

1. It’s way cooler to get your HDXPRT DVDs in person! That’s right: Bill’s not just going to show you the benchmark, he’s going to give you your very own copy! He’ll probably shake your hand, too, and thank you for coming. Admit it: that’s cooler than getting it in the mail (which is also pretty darn good—and which will happen to you if you join the HDXPRT Development Community).

Mark Van Name

Comment on this post in the forums

Posted in HDXPRT Development Community benefits |

An example of the community in action

By Mark Van Name

on April 27, 2011

Last week, I hosted a Webinar on HDXPRT. We’ll make a recording of it available on the site fairly soon. Multiple members attended. As I was going through the slides and discussing various aspects of the benchmark, a member asked about installing the benchmark from a USB key or a server. My response was the simple truth: we hadn’t considered that approach. As I then elaborated, we clearly should have thought about it, because those capabilities would be useful in just about every production lab out there, including ours here at PT. I concluded by saying that we’d look into it.

I’m not naming the member simply because with big companies I’m never sure if doing that will be good or will cause someone trouble, and I don’t want to cause hassle for anyone. He should, though, feel free to step forward and claim the well-deserved credit for the suggestion.

Less than a week after the Webinar, I’m happy to be able to report that the team has done more than look into these capabilities; it’s implemented them! So, the next Beta release, Beta 2, which we’ll be releasing any time now (maybe even before we post this blog entry), lets you install the benchmark from a network share or a USB key.

I know this is a relatively small thing, but I think it bears reporting because it is exactly the way the community should work. A member brought the benefits of his experience to bear in a great bit of feedback, and now the benchmark is better for it—and so are all of us who use it.

Keep the good ideas coming!

Mark Van Name

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT capabilities, HDXPRT Development Community benefits, HDXPRT development process, HDXPRT Webinar | Tagged Beta, Community |

Welcome!

By Bill Catchings

on April 4, 2011

Welcome to the HDXPRT Development Community blog! I’m Bill Catchings, and Mark Van Name and I will be writing this blog along with a few other key folks from the PT (Principled Technologies) team. I’ve worked on benchmarks and performance analysis since the 1980s. On the team at PT, we have people who’ve worked on such influential past benchmarks as Winstone, WinBench, 3D WinBench, and WebBench. We’re all excited to be leading the HDXPRT Development Community (HDC).

Our goal with this blog is to give a glimpse of the world behind the HDXPRT (High Definition eXperience Performance Ratings Test) benchmark and its development community. HDXPRT 2011 will be the result of the work of the HDC to define what the benchmark should do. HDXPRT 2011 is now in beta test with members of the HDC. Over the next few weeks, we would greatly appreciate whatever help you can give us in trying out the benchmark, shaking out bugs, and producing a polished product.

Thanks in advance to the HDC!

If you are reading this and are not a member of the HDC, I’d like to encourage you to join us. Annual membership is just $20. If you join in the next week or so, you will get a copy of the HDXPRT 2011 Beta and be able to provide feedback on it. We’d love to have you on board!

Bill

Comment on this post in the forums

Posted in HDXPRT Development Community benefits, HDXPRT development process |

Category: HDXPRT Development Community benefits

Who is on board?

What to do with all the times

Getting to the source

Top 5 reasons for meeting us at Computex in Taipei

An example of the community in action

Welcome!

Check out the other XPRTs: