BenchmarkXPRT Blog banner

Category: HDXPRT

What to do with all the times

HDXPRT, like most other application-based benchmarks, works by timing lots of individual operations. Some other benchmarks just time the entire script. The downside of that approach is that the time includes things that are constant regardless of the speed of the underlying hardware. Some things, like how fast a menu drops down or text scrolls, are tied to the user experience and should not go faster on a faster system. Including those items in the overall time dilutes the importance of the operations that we wait on and are frustrated by, the operations we need to time.

In the case of HDXPRT 2011, we time between 20 and 30 operations. We then roll these up into the times we report as well as the overall score. We do not, however, report the individual times. We expect to include even more timed operations in HDXPRT 2012. As we have been thinking about what the right metrics are, we have started to wonder what to do with all of those times. We could total up the times of similar operations and create additional results. For example, we could total up all the application load times and produce an application-load result. Or, we could total up all the times for an individual application and produce an application result. I can definitely see value in results like those.

Another possibility is to try and look at the general pattern of the results to understand responsiveness. One way would be to collect the times in a histogram, where buckets correspond to ranges of response times for the operations. Such a histogram might give a sense of how responsive a target system feels to an end user. There are certainly other possibilities as well.

If nothing else, I think it makes sense to expose these times in some way. If we make them available, I’m confident that people will find ways to use them. My concern is the danger of burdening a benchmark with too many results. The engineer in me loves all the data possible. The product designer knows that focus is critical. Successful benchmarks have one or maybe two results. How to balance the two?

One wonder of this benchmark development community is the ability to ask you what you think. What would you prefer, simple and clean or lots of numbers? Maybe a combination where we just have the high-level results we have now, but also make other results or times available in an “expert” or an “advanced” mode? What do you think?

Bill

Comment on this post in the forums

Turning a role into a scenario

To give you an idea about our process for proposing the scenarios in the RFC, I want to walk you through one of them. (Download the RFC at http://www.hdxprt.com/forum/hdxprt2012RFC.php if you haven’t already.) The role I’m going to discuss is that of a photo blogger. By photo blogger, I am thinking of someone who is a photo enthusiast, not someone who snaps pictures with his iPhone and puts them on Facebook. By my thinking, this type of photo blogger is someone who enjoys photography, has substantial knowledge of digital tools, and is interested in processing and posting photos on enthusiast Web sites. That person would commonly use raw rather than compressed photos from their camera and convert them to a standard format as the last step. The person would certainly edit the photos using different filters and effects. That person might also do some more esoteric things like stitch together multiple photos to create a panorama or work with high dynamic range (HDR) images. The latter is growing in popularity due to its ability both to better represent the color palette found in nature and to go farther and create amazing surreal images. Some newer cameras (like my Sony NEX-5) will produce the necessary multiple exposures to make HDR fairly easy to do.

What software would such a person use to do those things? Because HDXPRT is a consumer-oriented benchmark, we want to find affordable (or free) applications that a typical consumer might use. For photo manipulation, I think Adobe Photoshop Elements is the leading consumer application. The software lets users create, edit, organize, and share images. For HDR images, HDRsoft Photomatix is a good tool for high-quality two-stage HDR processing. It provides capabilities for both image overlaying and adjustable tone mapping. Another useful tool is GIMP, which stands for GNU Image Manipulation Program. It’s a free graphics editor that allows users to edit and retouch images. I think including some free tools like this mimics what many consumers with limited budgets are using.

Having seen the basic process we used in the RFC, what do you think? Does that process make sense to you? In this particular one, do you agree with the basic activities? The applications? Thanks, I look forward to hearing from you!

Bill

Comment on this post in the forums

HDXPRT 2012 RFC

We are putting the finishing touches on the RFC for HDXPRT 2012 and will make it available today. We are using the top-down approach I discussed last week, including the six roles I proposed. Our major objective with the RFC is to get your feedback. That feedback played an important part in developing HDXPRT 2011, and we are hoping it plays an even larger role in developing HDXPRT 2012.

The RFC, or request for comments, includes our thoughts and ideas for the design of HDXPRT 2012 based on the many conversations we’ve had over the six months since the current version of HDXPRT debuted. At this point, nothing is written in stone. Now is the time to let us know where you agree and where you disagree.

The RFC is available for Development Community members at http://www.hdxprt.com/forum/hdxprt2012RFC.php. Our goal is to get your feedback by December 2. We’d like as much of the feedback as possible to appear on the forums to help stimulate discussion. However, if you prefer to send in your comments via email, please send them to HDXPRTsupport@hdxprt.com.

Regardless of the mechanism, we want and need your feedback. Thanks! I look forward to hearing from you!

Bill

Comment on this post in the forums

How do you use high-definition media?

The first step in our top-down process of defining the HDXPRT 2012 is to look at what people actually do today with high-definition media. The obvious place to start is with what we do with photos, video, and audio. Or, in my case, with what I do.

I regularly do about six different things with high-definition media:

  1. Organize media – This is something I do often. Maybe I’m more of a curator than a creator! In this category I include everything from keeping track of media, to doing simple enhancements (for example, red eye removal from photos), to converting to different formats.
  2. Create media – I include here not only capturing the media, but also manipulating photos and videos. These usages may not be the most difficult ones in the applications, but we all do them, and we all wait on them.
  3. Photo blog – This covers the fancier photo work, typically with more complex editing using digital tools.
  4. Produce video – When you really work on a video, you end up with editing and manipulation tasks that really stress your system. I often end up waiting for my computer to finish this sort of work.
  5. Create music – I’m not very good at this type of work, but mixing, editing, remixing, and sharing music can be a lot of fun!
  6. Viewing video – Finally, we come to viewing video, which is basically watching videos in different HD formats. I’m much better at this.

Some tasks I do often, such as looking at photos or listening to music, don’t tend to be stressful enough on a computer to be worth measuring in the benchmark. Consequently, I didn’t include them here.

That’s just my take, though.

What do you do? What tasks would you like to see in HDXPRT 2012?

Bill

Comment on this post in the forums

An open, top-down process

We’ve been hard at work putting together the RFC for HDXPRT 2012. As a group of us sat around a table discussing what we’d like to see in the benchmark, it became clear to me how different this development process is from those of other benchmarks I’ve had a hand in creating (3D WinBench, Winstone, WebBench, NetBench, and many others.). The big difference is not in the design or the coding or even the final product.

The difference is the process.

A sentiment that came up frequently in our meeting was “Sure, but we need to see what the community thinks.” That indicates a very different process than I am used to. Different from what companies developing benchmarks do and different from what benchmark committees do. What it represents, in a word, is openness. We want to include the Development Community in every step of the process, and we want to figure out how to make the process even more open over time. For example, we discussed ideas as radical as videoing our brainstorming sessions.

Another part of the process I think is important is that we are trying to do things top-down. Rather than deciding which applications should be in the benchmark, we want to start by asking how people really use high-definition media. What do people typically do with video? What do they do to create it and how do they watch it? Similarly, what do people do with images and audio?

At least as importantly, we don’t want to include only our opinions and research on these questions; we want to pick your brains and get your input. From there, we will work on the workflows, the applications, and the RFC. Ultimately, that will lead to the scripts themselves. With your input and help, of course!

Please let us know any ideas you have for how to make the process even more open. And tell us what you think about this top-down approach. We’re excited and hope you are, too!

Bill

Comment on this post in the forums

Discussing the future

As I mentioned a couple weeks back, we’ve begun the HDXPRT 2012 development cycle by asking for suggestions. Before we get much further in the process, we’d like an opportunity to talk with you more directly. The best way to do that is via a webinar.

I’ll be hosting an HDXPRT webinar this coming Friday, October 14 at 2:00pm EST. I’m hoping you’ll be able to join us. I’ll go over the progress we’ve made this year in the HDXPRT Development Community, talk about the HDXPRT 2011 source code we recently released to members, and discuss the roadmap for the development cycle of HDXPRT 2012 that is just starting.

We really want your feedback on the current benchmark and your input on future directions. I encourage you to attend the webinar and let us know your ideas and suggestions. I expect the webinar will last about 45 minutes, depending upon the questions people have. We will be sending out email invitations to members shortly. If you have not joined the community, please do so now (at http://www.hdxprt.com/forum/register.php) and we will get you an invitation as well.

We know you may be unable to make the scheduled time, so we’ll post the webinar when it is over, as we have done in the past (http://hdxprt.com/webinar).

We value your input and participation in the HDXPRT benchmark process and look forward to your joining us later this week for the webinar. “See” you there!

Bill

Comment on this post in the forums

Check out the other XPRTs:

Forgot your password?