Author Archives: Bill Catchings

Now it’s your turn

on June 1, 2012

We mailed the DVDs for the HDXPRT 2012 this week. Kudos to the development and test teams. Thanks to all of the community members who confirmed their addresses. You should receive the DVDs soon. Now the fun begins.

The comment period for the beta goes through June 29. Your feedback is essential. Please install the benchmark, run it, and let us know what you think. If you have questions, we’ll answer them as quickly as possible. If you find a problem, we want to fix it. If there’s something you don’t like, we want to know. If there’s something you do like, we’d like to know that too.

If you don’t receive your DVDs next week, please contact us and we’ll make sure you get your copy ASAP. Unfortunately, the HDXPRT 2012 beta is not available for download. As we explained in an earlier blog post, the benchmark is approximately 10 GB.

Remember, the end of the comment period is June 29. Please help us make HDXPRT 2012 the best possible benchmark.

Eric

Comment on this post in the forums

HDXPRT 2012 and Windows 8

By Bill Catchings

on May 25, 2012

One of our community members asked whether HDXPRT 2012 would support Windows 8. It certainly is one of our goals. The current beta does not work with Windows 8, however. Problems with a couple of the applications themselves prevent the current version from installing on Windows 8.

We will continue to test beta versions of HDXPRT 2012 with beta versions of Windows 8 over the coming weeks. There is rumored to be a newer beta of Windows 8 out next week and we are hoping to be able to resolve the issues in a later beta or at least by the shipping version.

Regardless, there will be multiple versions of Windows 8 before it releases later this year. We plan to release HDXPRT 2012 this summer. We assume there will be some glitches. So, we expect that we will have to release an update to HDXPRT 2012 once Windows 8 is closer to its final version.

Also, once there is a version of Windows 8 that allows publishing performance results, we plan to look at what performance differences there are between Windows 7 and Windows 8.

A quick update on the beta—we are in the process of making the DVDs and will send them out next week. We sent emails to all the community members to confirm their correct shipping addresses. If you received that email, please respond. If you haven’t received that email, please let us know. We want to make sure the DVDs actually get to you!

Bill

Comment on this post in the forums

Today is a milestone

By Bill Catchings

on May 18, 2012

Today is a milestone. We are freezing the beta for HDXPRT 2012. It’s taken a lot of work–developing something with this many moving parts is a bit like herding cockroaches. However, the workloads are set and the benchmark seems stable.

So, what happens next?

First, the beta goes to a team who will spend a few days intensively testing it. The test team is isolated from the development team because, let’s face it, developers can’t test their own code. (I should know. I’ve developed a lot of products in my time.)

We’ll run it on a variety of systems, checking it for stability and repeatability. We will also look at the results and make sure that they are sensible. That is, they reflect that capabilities of the systems we are running the benchmark on.

The beta is approximately 10 GB. It includes installers for nine applications and a number of large content files. Because of its size, the beta will only be available on DVD. Once the tests look good, we’ll start reproducing and verifying the DVDs. Once the DVDs are verified, we’ll mail them out. The whole process should take about a week, so community members should have their beta DVDs within a couple of weeks.

While the testing is happening, we’ll be contacting each community member to confirm that we have their correct physical mail address.

Once you have the DVDs, please run the benchmark and send feedback. We need your help to make this beta into a truly great release candidate. We will work hard to address any feedback we get. If you find a problem, let us know. If you have a complaint, let us know. It you have a compliment, by all means, let us know!

Next week, Bill will be back from his travels.

Eric

Comment on this post in the forums

Posted in HDXPRT 2012 beta, HDXPRT development process, Let us know your thoughts |

More HDXPRT 2012 changes

By Bill Catchings

on May 11, 2012

We’re in the home stretch of testing the HDXPRT 2012 before releasing the beta to the community members. As is often the case, we’ve run into some issues that I want to share with you.

We’ve encountered some technical challenges in creating the 4K video playback workload that the design specification called for. We could not use the existing HDXPRT playback automation and instrumentation because Windows Media Player currently does not by default play 4K H.264 content. We tried other video players but ran into stability issues on slower systems.

These technical problems led us to step back and think about the role of video playback in HDXPRT 2012. In HDXPRT 2011, there wasn’t much differentiation between most current systems on the video playback results. The video playback tests were also a source of confusion. As I traveled and talked to users, they thought that the stars of the video playback were the primary metric, not the performance number. More generally, however, having both quality and performance numbers can be confusing.

Our plan is to remove the video playback tests from the beta. We will include them as optional inspection tests in an updated version later this year. By removing video playback from the primary tests and metrics, we’ll be able to include stressful test that won’t work on all systems while avoiding unnecessary confusion. Over time, I think we can include a number of interesting tests this way. We’d love to hear your ideas for what might be good to include that way as well as whether you think we’re making the correct call for video playback tests in HDXPRT 2012.

I will be traveling next week. Eric Hale, our project lead for HDXPRT 2012, will be writing the next blog. He will hopefully have some good news about the beta!

Bill

Comment on this post in the forums

Posted in HDXPRT 2012 beta, HDXPRT development process |

HDXPRT 2012 characterization study

By Bill Catchings

on May 4, 2012

For HDXPRT 2011, we did quite a bit of testing to characterize the benchmark. Those results appeared in an initial white paper and a follow up one. In the first, we ran tests on different processors, various amounts of RAM, internal vs. external graphics, hard disk vs. SSD, and the effects of Intel Turbo Boost Technology. In the follow up white paper, we looked in more depth at the effect of graphics cards with different processors.

I mentioned a couple weeks ago that we are starting to put together a testbed to help us characterize HDXPRT 2012. What is in that testbed will define what characteristics of the upcoming benchmark we measure. We would like to get your help defining that testbed.

Our current thinking is to do a similar set of tests this year with updated hardware. However, plenty of additional things would be interesting to look at. First, I would like to increase the range of processors we test, including AMD processors. I would also like to do some testing varying different processor characteristics such as threads, cores, and frequency. It might also be good to look at the effect of new technologies like hybrid drives (which combine a small SSD with a hard disk to try and have the best of both).

We face two challenges in doing these characterization tests. One is to try and change one only variable at a time. That is very difficult in some cases, such as comparing Intel and AMD processors—you can’t just swap them in the same motherboard. Fortunately, it is usually possible to find very similar motherboards and keep other components (like disks, graphics, and RAM) constant. The other challenge is getting all of the necessary hardware in house.

So, we have two requests for you. First, let us know what you would like to see us test. Second, help us by supplying some of that equipment. If you supply the equipment we will do our best to include results from it in the characterization study and in the new HDXPRT 2012 results database. As always, thanks for your help!

Bill

Comment on this post in the forums

Posted in HDXPRT 2012, HDXPRT development process, HDXPRT workloads, Let us know your thoughts |

Change is inevitable

By Bill Catchings

on April 27, 2012

As we get close to the beta version of HDXPRT 2012, I wanted to let you know how it compares with the original design specification. As inevitably happens in any software project, there are differences between the original design and the final product. Generally, things have stayed pretty close with HDXPRT 2012, but there are two changes worth noting.

First, in the design specification, we specified Audacity 1.3.14 Beta in the Music Maker scenario as that was the only version that supported Windows 7 at the time. Audacity 2.0 with Windows 7 debuted in the interim so we are using that version.

The second and more significant issue was with Picasa, which was to be part of the Media Organizer scenario. Unfortunately, we couldn’t create a stable script because scripting tools like AutoIT could not properly recognize some of Picasa’s application UI elements. Somewhat reluctantly, we ended up replacing Picasa with Photoshop Elements. We still think the scenario is a good one and Photoshop Elements is an appropriate tool. I would have liked, however, to have Picasa in there.

There are probably some more minor differences between the beta and the design specification. We’ll let you know what they are when we have the beta ready in a couple weeks. (Hopefully!) We’re looking forward to getting that into your hands and getting your feedback. If you’re not already a member of the Development Community, I encourage you to join so that you can get your copy of the beta when it is available.

Bill

Comment on this post in the forums

Posted in HDXPRT 2012, HDXPRT 2012 design specification, HDXPRT development process |

Author Archives: Bill Catchings

Now it’s your turn

HDXPRT 2012 and Windows 8

Today is a milestone

More HDXPRT 2012 changes

HDXPRT 2012 characterization study

Change is inevitable

Check out the other XPRTs: