Category: HDXPRT

Loose ends

on March 21, 2013

As we mentioned last week, the Samsung debuted the Galaxy S4 this past week. It seems to have hit all the expectations – including eye-scrolling. Looking at the reviews, it’s somewhere between the greatest phone ever made and something not quite as good as the iPhone. I’m looking forward to seeing one for myself and hoping someone submits a WebXPRT score for it soon.

We’ll be releasing the HDXPRT 2013 design document tomorrow. As we’ve said, the number one comment has been that HDXPRT 2012 is too big and takes too long to run. We have put a lot of thought into how to trim HDXPRT 2013 and still keep the essential value of the benchmark. We’ve also received some other good feedback that we’re incorporating, such as making installing and running HDXPRT scriptable.

We’ve been doing some investigation during the RFC period, and we’ve encountered problems scripting some of the applications, notably iTunes and PowerDirector. We’re working to overcome these problems but if they prove to be insurmountable, we might have to change the list of applications.

Again, thanks to everyone who commented on the HDXPRT 2013 RFC.

We’ve learned that some WebXPRT users in mainland China are having problems with very slow downloads. The good news is that results from the runs are valid. However, we understand that this is frustrating and are investigating solutions. If you are in China and have experienced slow downloads, please send an e-mail to benchmarkxprtsupport@principledtechnologies.com. We would like your help in evaluating any solutions we come up with.

Speaking of WebXPRT, over the next few weeks we’ll be releasing several white papers that look at WebXPRT results in more depth. The first paper will explain the WebXPRT confidence interval and how it relates to run to run variability, which has been confusing to a number of people. The second paper will look at the effect on the browser on WebXPRT scores. The third paper will look at the influence of the operating system.

Eric

Comment on this post in the forums

Posted in HDXPRT 2012 RFC, HDXPRT 2013, HDXPRT 2013 design document, WebXPRT, WebXPRT 2013 results |

Soon, we’ll know the truth

By Eric Hale

on March 14, 2013

I’m writing this on the morning of one of the most anticipated events in tech, Samsung’s rollout of the new Galaxy S4 phone. There have been many leaks, and it will be interesting to see how many are true. One feature generating buzz would let you scroll just by moving your eyes. Samsung filed for the patent, but there are conflicting reports about whether eye scrolling made it into the Galaxy S4.

The leaked reports say that the U.S. version of the Galaxy S4 will use Qualcomm Inc.’s quad-core chip, while versions for other markets will use Samsung’s Exynos 5 Octa-core chip. As we continue development of PhoneXPRT, I will be very interested to see how the different processors stack up performance wise.

For browser-level performance, WebXPRT 2013 is available today. If you get a new Galaxy S4, try out WebXPRT 2013 on it. As it says on the WebXPRT.com page, your run is confidential. However, anyone who shares a Galaxy S4 score in the next week gets a free t-shirt (until they are gone). You can tag BenchmarkXPRT on a Facebook post, use the hashtag #WebXPRT on Twitter, or email us at BenchmarkXPRTsupport@principledtechnologies.com.

In other news, the HDXPRT 2013 comment period has ended and we’re working on the design document. We’ll be sharing that with the community next week. If you have comments you haven’t sent in, please do so, and we’ll try to address them in the design document.

Eric

Comment on this post in the forums

Posted in HDXPRT 2012 RFC, HDXPRT 2013, PhoneXPRT, WebXPRT |

No time to rest

By Eric Hale

on March 7, 2013

Last week was possibly our biggest week ever. We announced PhoneXPRT, a new benchmark for evaluating the performance of smartphones, and released TouchXPRT 2013 and WebXPRT 2013 to the general public.

Since then, there’s been a lot of interest. The numbers keep going up! We’re not just talking about page views – people are downloading TouchXPRT and lots of people are running WebXPRT.

People have also been downloading the TouchXPRT source, which is very exciting. We strongly encourage community members to look at how the benchmark is put together. If you have programming skills and want to submit code, get in touch with us at benchmarkxprtsupport@principledtechologies.com. We’d love to hear from you!

Oh, and there’s a new video! This one introduces WebXPRT to the public. It gives a good idea of the range of devices WebXPRT will run on.

That’s a lot for one week. We’re not resting on our laurels, though. Obviously, we’re working on PhoneXPRT. However, let’s not forget about HDXPRT. The comment period for HDXPRT 2013 officially closed on March 6 and we are starting to work on the HDXPRT 2013 design document. If you have any feedback you haven’t sent, please do send it on. We’ll do our best to incorporate it into the design document.

Eric

Comment on this post in the forums

Posted in HDXPRT 2013, HDXPRT 2013 RFC, PhoneXPRT, TouchXPRT 2013 release, TouchXPRT 2013 source, Video, WebXPRT, WebXPRT 2013 release |

The HDXPRT 2013 RFC is here

By Eric Hale

on February 14, 2013

We released the RFC, or request for comments, for HDXPRT 2013 yesterday. Our major objective with the RFC is to get your feedback. Your feedback played an important part in developing HDXPRT 2012, and we are hoping it plays an even larger role in developing HDXPRT 2013.

The RFC includes our thoughts and ideas for the design of HDXPRT 2013 based on the many conversations we’ve had over the six months since the current version of HDXPRT debuted. Indeed, during the last few weeks, we shared some of the feedback we received during and after the Webinar in January.

At this point, nothing is written in stone. Now is the time to let us know where you agree and where you disagree. For example, the current proposal drops support for Windows 7. Do you have an opinion about this? Let us know.

The RFC is available for Development Community members at http://www.principledtechnologies.com/hdxprt/forum/hdxprt2013RFC.php. Our goal is to get your feedback by March 6. We’d like as much of the feedback as possible to appear on the forums to help stimulate discussion. However, if you prefer to send in your comments via email, please send them to BenchmarkXPRTsupport@hdxprt.com.

Of course, you can send comments to us any time, and you don’t have to limit yourself to HDXPRT! Do you have thoughts about TouchXPRT or WebXPRT? They are both moving rapidly toward their official releases. Do you have thoughts about other benchmarks we should consider developing? Send those, too!

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT 2013, HDXPRT 2013 RFC, HDXPRT Development Community benefits, HDXPRT development process |

Straight from the source

By Eric Hale

on February 7, 2013

One of the pillars of our community model of benchmark development is making the source available. As we’ve said many times, we believe that doing so leads to better benchmarks.

Today we released the source for HDXPRT 2012. As with previous versions of HDXPRT, the source is available only to community members, not to the general public. We apologize that it has taken so long. HDXPRT is complicated to build, and we wanted to have a simpler and more robust build process before we made the source available.

The source allows you to examine how HDXPRT is implemented and to try some experiments of your own. Because of the size of HDXPRT 2012, the source package does not include the applications or the data files for the workloads. By including only the benchmark source code and associated files, we could keep the package small enough to download. If you want to try some changes for experiments and test them, all you need to do is install HDXPRT 2012 from the distribution DVDs. The compilation instructions will tell you how to copy your modified executables over the shipping versions.

Community members can get instructions on how to download the source code here (registration required).

If you create something interesting while you’re experimenting, let us know! We’d love to have the community consider it for HDXPRT 2013.

Speaking of the community, we’ve sent T-shirts to all community members who’ve supplied their up-to-date mailing address. If you’re a community member who wants a shirt but hasn’t yet let us know, please e-mail benchmarkxprtsupport@principledtechnologies.com with your mailing address by February 15th.

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT, HDXPRT Development Community benefits, HDXPRT source code |

Keep them coming!

By Eric Hale

on January 31, 2013

Questions and comments have continued to come in since the Webinar last week. Here are a few of them:

How long are results valid? For a reviewer like us, we need to know that we can reuse results for a reasonable length of time. There is a tension between keeping results stable and keeping the benchmark current enough for the results to be relevant. Historically, HDXPRT allowed at least a year between releases. Based on the feedback we’ve received, a year seems like a reasonable length of time.
Is HDXPRT command line operable? (asked by a community member with a scripted suite of tests) HDXPRT 2012 is not, but we will consider adding a command line interface for HDXPRT 2013. While most casual users don’t need a command line interface, it could be very valuable to those of us using HDXPRT in labs.
I would be hesitant to overemphasize the running time of HDXPRT. The more applications it runs, the more it can differentiate things and the more interesting it is to those of us who run it at a professional level. If I could say “This gives a complete overview of the performance of this system,” that would actually save time. This comment was a surprise, given the amount of feedback we received saying that HDXPRT was too large. However, this gets to the heart of why we all need to be careful as we consider which applications to include in HDXPRT 2013.

If you had to miss the Webinar, it’s available at the BenchmarkXPRT 2013 Webinars page.

We’re planning to release the HDXPRT 2013 RFC next week. We’re looking forward to your comments.

Eric

Comment on this post in the forums

Posted in Benchmark metrics, Benchmarking, HDXPRT, HDXPRT workloads |

Category: HDXPRT

Loose ends

Soon, we’ll know the truth

No time to rest

The HDXPRT 2013 RFC is here

Straight from the source

Keep them coming!

Check out the other XPRTs: