Category: Collaborative benchmark development

Straight from the source

on February 7, 2013

One of the pillars of our community model of benchmark development is making the source available. As we’ve said many times, we believe that doing so leads to better benchmarks.

Today we released the source for HDXPRT 2012. As with previous versions of HDXPRT, the source is available only to community members, not to the general public. We apologize that it has taken so long. HDXPRT is complicated to build, and we wanted to have a simpler and more robust build process before we made the source available.

The source allows you to examine how HDXPRT is implemented and to try some experiments of your own. Because of the size of HDXPRT 2012, the source package does not include the applications or the data files for the workloads. By including only the benchmark source code and associated files, we could keep the package small enough to download. If you want to try some changes for experiments and test them, all you need to do is install HDXPRT 2012 from the distribution DVDs. The compilation instructions will tell you how to copy your modified executables over the shipping versions.

Community members can get instructions on how to download the source code here (registration required).

If you create something interesting while you’re experimenting, let us know! We’d love to have the community consider it for HDXPRT 2013.

Speaking of the community, we’ve sent T-shirts to all community members who’ve supplied their up-to-date mailing address. If you’re a community member who wants a shirt but hasn’t yet let us know, please e-mail benchmarkxprtsupport@principledtechnologies.com with your mailing address by February 15th.

Eric

Comment on this post in the forums

There is such a thing as too much

By Eric Hale

on December 6, 2012

There’s been a lot of excitement about TouchXPRT recently. However, we haven’t been ignoring HDXPRT. On November 9, we released a patch that lets HDXPRT support Windows 8. We’ve now integrated the patch into HDXPRT2012, so all copies of HDXPRT 2012 going forward will install on Windows 8 without the need for a separate step.

As promised, we will be releasing the source code for HDXPRT 2012. We anticipate having it available for community members by December 14.

During the comment period for HDXPRT, this message came through loud and clear: HDXPRT 2012 is too big and takes too long to run. So we are working hard to find the best way to reduce the number of applications and scenarios. While we want to make the benchmark smaller and faster, we want to make sure that HDXPRT 2013 is comprehensive enough to provide useful performance metrics for the greatest number of people.

We’re working toward having an RFC in late January that will define a leaner, meaner HDXPRT 2013, and will reflect the other comments we have as received as well. If you have thoughts about which applications and scenarios are most important to you, please let us know.

In other news, CES is coming in January, and Principled Technologies will be there! Once again, Bill is hoping to meet with as many of you in the Development Community as possible. We’ll have a suite at the Hilton and would love for you to come, kick back, and talk about HDXPRT, TouchXPRT, the future of benchmarks, or about the cool things you’ve seen at the show. (Bill loves talking about gadgets. Last year, he went into gadget overload!)

If you plan to be at CES, but are stuck working a booth or suite, let us know and Bill will try to stop by and say hi. Drop us an email at hdxrpt_CES@principledtechnologies.com and we will set up an appointment.

Finally, we’re really excited about the big changes at the Principled Technologies Web site. The new Web site gives us a lot of opportunities. Over the next few weeks, we’ll be looking at ways the Development Community can take advantage of them.

Eric

on this post in the forums

Posted in Application-based benchmarks, Benchmarking, CES, Collaborative benchmark development, HDXPRT 2013, HDXPRT Development Community benefits, Principled Technologies, What makes a good benchmark?, Windows 8 | Tagged CES |

TouchXPRT in the fast lane

By Bill Catchings

on October 25, 2012

I titled last week’s blog “Putting the TouchXPRT pedal to the metal.” The metaphor still applies. On Monday, we released TouchXPRT 2013 Community Preview 1 (CP1). Members can download it here.

CP1 contains five scenarios based on our research and community feedback. The scenarios are Beautify Photo Album, Prepare Photos for Sharing, Convert Videos for Sharing, Export Podcast to MP3, and Create Slideshow from Photos.

Each scenario gives two types of results. There’s a rate, which allows for simple “bigger is better” comparisons. CP1 also gives the elapsed time for each scenario, which is easier to grasp intuitively. Each approach has its advantages. We’d like to get your feedback on whether you’d like us to pick one of those metrics for the final version of TouchXPRT 2013 or whether it makes more sense to include both. You’ll find a fuller description of the scenarios and the results in the TouchXPRT 2013 Community Preview 1 Design overview.

While you’re looking at CP1, we’re getting the source ready to release. To check out the source, you’ll need a system running Windows 8, with Visual Studio 2012 installed. We hope to release it on Friday. Keep your eye the TouchXPRT forums for more details.

Post your feedback to the TouchXPRT forum, or e-mail it to TouchXPRTSupport@principledtechnologies.com. Do you want more scenarios? Different metrics? A new UI feature? Let us know! Make TouchXPRT the benchmark you want it to be.

As I explained last week, we released CP1 without any restrictions on publishing results. It seems that AnandTech was the first to take advantage of that. Read AnandTech’s Microsoft Surface Review to see TouchXPRT in action.

We are hoping that other folks take advantage of CP1’s capability to act as a cross-platform benchmark on the new class of Windows 8 devices. Come join us in the fast lane!

Bill

Comment on this post in the forums

Posted in Benchmark metrics, Collaborative benchmark development, Touch-based benchmarking, Touch-based devices, TouchXPRT development process, TouchXPRT release cycle, TouchXPRT results |

Make sure your voice is heard

By Eric Hale

on October 4, 2012

One thing about the community model we use for developing HDXPRT is that is depends on the community. Your input is essential to making the benchmark the best it can be. As the community grows, we’re learning more about your priorities.

During the development of HDXPRT 2012, we made the decision to remove the playback tests from the benchmark. While the design document called for the playback test to include 4K H.264, Windows Media Player does not play that format by default. Because less demanding codecs were not differentiating systems, and because the stars used to report the results confused some people, it seemed like a reasonable decision. Bill announced the decision in a blog post, More HDXPRT 2012 changes.

Fast forward to September 18, when Bill hosted the HDXPRT 2012 Webinar. During the Q&A session, a new member of the community said that the playback tests from HDXPRT 2011 were what got him interested in the benchmark. For now, he has to use HDXPRT 2011 for those tests, although, as per Bill’s original blog post, we may release a more demanding playback test as a standalone inspection test later this year.

The suggestion period for HDXPRT 2013 started on October 1. Now is the time to let us know what tests are the most useful to you. If there are tests you’d like us to add, tests you’d like us to change, applications you’d like us to consider, we need to know that too. You can post your suggestions to the forum in the HDXPRT 2013 Suggestions section or mail them to hdxprtsupport@hdxprt.com.

In November, we’ll develop an RFC for HDXPRT 2013 and send it to the community for review.

While the suggestions we receive early have the best chance of being implemented, comments we receive after the formal suggestion period still get our attention. We’re always listening. Contact us anytime and make sure that HDXPRT 2013 includes the things that are important to you.

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT 2012, HDXPRT 2013, HDXPRT Webinar, Let us know your thoughts |

Meeting the old fashioned way

By Mark Van Name

on August 30, 2012

As Bill discussed last week, we work in many different ways to build the community. Facebook, Webinars, Twitter, this blog—all of these social media venues help connect those of us in the community.

Sometimes, though, it’s nice to sit down in person and chat.

Which is precisely what I’m inviting fellow community members to do at the upcoming Intel Developer Forum (IDF). IDF runs from September 11 to 13 at the Moscone West Convention Center in San Francisco. While I’ll be busy attending the show for part of each day, I’d love to spend some of my free time talking with people about both HDXPRT and TouchXPRT. We could do it informally, by simply running across each other on the show floor, or we could schedule some meetings.

We could even make a group lunch of it. If a few folks can agree on a day and time, I’d be happy to set up and host a lunch at a nearby restaurant so we can share ideas while enjoying a meal.

If you’re interested, feel free to email me, and we’ll arrange to get together. I hope to have many good conversations with community members at the show!

Mark Van Name

Comment on this post in the forums

Posted in Benchmarking, Collaborative benchmark development, Let us know your thoughts |

Growing the community

By Bill Catchings

on August 23, 2012

Growing the community

It’s probably stating the obvious to say that what makes the XPRT Development Community work is the people. The more people there are in the community, the better the resulting benchmarks will be. Naturally, that means we want more people in the community.

Part of PT’s role in building the community is to get more people involved. Over the last couple of years, that has meant doing a number of different things—from conducting Webinars (we’ll have the next one in September) to meeting at trade shows (we expect to be at CES again next year) to contacting individuals in person and via email to using social media like Twitter (follow us at @HDXPRT) and Facebook (like HDXPRT Development Community). As I mentioned last week, we are now working on some videos to help get out the message. Basically, we’ve been telling everyone we can about the benchmarks and the community that creates them.

One thing we haven’t done, however, is to actively enlist the community itself in building the community. Here are some ways you can help. Please like, follow, or whatever us on the different social media platforms. Invite others to join the community. Probably most importantly, use the benchmarks and publish the results. Now that HDXPRT 2012 is out, we want to see its results popping up in as many places as possible.

You can also help by telling us your ideas for how to grow the community. What else do you think we should be doing?

Bill

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT 2012 |

Category: Collaborative benchmark development

Straight from the source

There is such a thing as too much

TouchXPRT in the fast lane

Make sure your voice is heard

Meeting the old fashioned way

Growing the community

Check out the other XPRTs: