BenchmarkXPRT Blog banner

Author Archives: Eric Hale

Lies, damned lies, and statistics

No one knows who first said “lies, damned lies, and statistics,” but it’s easy to understand why they said it. It’s no surprise that the bestselling statistics book in history is titled How to Lie with Statistics. While the title is facetious, it is certainly true that statistics can be confusing—consider the word “average,” which can refer to the mean, median, or mode. “Mean average,” in turn, can refer to the arithmetic mean, the geometric mean, or the harmonic mean. It’s enough to make a non-statistician’s head spin.

In fact, a number of people have been confused by the confidence interval WebXPRT reports. We believe that the best way to stand behind your results is to be completely open about how you crunch the numbers. To this end, we released the white paper WebXPRT 2013 results calculation and confidence interval this past Monday.

This white paper, which does not require a background in mathematics, explains what the WebXPRT confidence interval is and how it differs from the benchmark variability we sometimes talk about. The paper also gives an overview of the statistical and mathematical techniques WebXPRT uses to translate the raw timing numbers into results.

Because sometimes the devil is in the details, we wanted to augment our overview by showing exactly how WebXPRT calculates results. The white paper is accompanied by a spreadsheet that reproduces the calculations WebXPRT uses. If you are mathematically inclined and would like to suggest improvements to the process, by all means let us know!

Eric

Comment on this post in the forums

Rapid Evolution

As Bill mentioned last week, we are considering changing the name of PhoneXPRT, not only because of the interest in using the benchmark scenarios on Android based tablets, but also because the line between phones and tablets has become blurred.

Devices that are too big to be a phone and too small to be a proper tablet are everywhere. PC Magazine says the first true “phablet” – possibly the ugliest portmanteau in the history of technology – was the AT&T EO 440 in 1993. However, it was more 8 years before Samsung had the first really successful phablet, the Galaxy Note. Now, less than 2 years later, there are rumors that Samsung may kill the Note in favor of the Samsung Galaxy Mega.

Obviously, this is one of the fastest evolving areas in tech. This rapid evolution has given us an almost bewildering array for devices, from small phones such as the Sony Xperia Mini, which a child can hold in one hand, to the ASUS Transformer AiO P1801, which has a whopping 18.4” screen! All this speed and diversity obviously pose challenges for the new benchmark, but it makes the work very exciting as well!

We have received comments about the name and we really appreciate those. If you have any thoughts, let us know. We hope to make a decision about whether to change the name soon.

Eric

P.S. I should note: The Transformer AiOP1801 also runs Windows 8, which means it’s a great candidate for TouchXPRT as well.

 

Comment on this post in the forums

Loose ends

As we mentioned last week, the Samsung debuted the Galaxy S4 this past week. It seems to have hit all the expectations – including eye-scrolling. Looking at the reviews, it’s somewhere between the greatest phone ever made and something not quite as good as the iPhone. I’m looking forward to seeing one for myself and hoping someone submits a WebXPRT score for it soon.

We’ll be releasing the HDXPRT 2013 design document tomorrow. As we’ve said, the number one comment has been that HDXPRT 2012 is too big and takes too long to run. We have put a lot of thought into how to trim HDXPRT 2013 and still keep the essential value of the benchmark. We’ve also received some other good feedback that we’re incorporating, such as making installing and running HDXPRT scriptable.

We’ve been doing some investigation during the RFC period, and we’ve encountered problems scripting some of the applications, notably iTunes and PowerDirector. We’re working to overcome these problems but if they prove to be insurmountable, we might have to change the list of applications.

Again, thanks to everyone who commented on the HDXPRT 2013 RFC.

We’ve learned that some WebXPRT users in mainland China are having problems with very slow downloads. The good news is that results from the runs are valid. However, we understand that this is frustrating and are investigating solutions. If you are in China and have experienced slow downloads, please send an e-mail to benchmarkxprtsupport@principledtechnologies.com. We would like your help in evaluating any solutions we come up with.

Speaking of WebXPRT, over the next few weeks we’ll be releasing several white papers that look at WebXPRT results in more depth. The first paper will explain the WebXPRT confidence interval and how it relates to run to run variability, which has been confusing to a number of people. The second paper will look at the effect on the browser on WebXPRT scores. The third paper will look at the influence of the operating system.

Eric

Comment on this post in the forums

Soon, we’ll know the truth

I’m writing this on the morning of one of the most anticipated events in tech, Samsung’s rollout of the new Galaxy S4 phone.  There have been many leaks, and it will be interesting to see how many are true. One feature generating buzz would let you scroll just by moving your eyes. Samsung filed for the patent, but there are conflicting reports about whether eye scrolling made it into the Galaxy S4.

The leaked reports say that the U.S. version of the Galaxy S4 will use Qualcomm Inc.’s quad-core chip, while versions for other markets will use Samsung’s Exynos 5 Octa-core chip. As we continue development of PhoneXPRT, I will be very interested to see how the different processors stack up performance wise.

For browser-level performance, WebXPRT 2013 is available today. If you get a new Galaxy S4, try out WebXPRT 2013 on it.  As it says on the WebXPRT.com page, your run is confidential. However, anyone who shares a Galaxy S4 score in the next week gets a free t-shirt (until they are gone). You can tag BenchmarkXPRT on a Facebook post, use the hashtag #WebXPRT on Twitter, or email us at BenchmarkXPRTsupport@principledtechnologies.com.

In other news, the HDXPRT 2013 comment period has ended and we’re working on the design document.  We’ll be sharing that with the community next week. If you have comments you haven’t sent in, please do so, and we’ll try to address them in the design document.

Eric

Comment on this post in the forums

No time to rest

Last week was possibly our biggest week ever. We announced PhoneXPRT, a new benchmark for evaluating the performance of smartphones, and released TouchXPRT 2013 and WebXPRT 2013 to the general public.

Since then, there’s been a lot of interest. The numbers keep going up! We’re not just talking about page views – people are downloading TouchXPRT and lots of people are running WebXPRT.

People have also been downloading the TouchXPRT source, which is very exciting. We strongly encourage community members to look at how the benchmark is put together. If you have programming skills and want to submit code, get in touch with us at benchmarkxprtsupport@principledtechologies.com.  We’d love to hear from you!

Oh, and there’s a new video! This one introduces WebXPRT to the public. It gives a good idea of the range of devices WebXPRT will run on.

That’s a lot for one week. We’re not resting on our laurels, though. Obviously, we’re working on PhoneXPRT. However, let’s not forget about HDXPRT. The comment period for HDXPRT 2013 officially closed on March 6 and we are starting to work on the HDXPRT 2013 design document. If you have any feedback you haven’t sent, please do send it on. We’ll do our best to incorporate it into the design document.

Eric

Comment on this post in the forums

What a week!

This has been quite a week for the BenchmarkXPRT family. We kicked off the week by announcing a new benchmark: PhoneXPRT. PhoneXPRT is designed to test small form factor devices, such as smart phones.  PhoneXPRT will initially run on Android. For more information, read the press release at http://www.prweb.com/releases/phoneXPRT/pt0213/prweb10461639.htm

There was more to come. Today we formally released TouchXPRT 2013 and WebXPRT 2013. The community and the media have been using these as community previews for weeks now. Now that we’ve released the benchmarks, anyone may freely use them.  You can read the press release at http://www.prweb.com/releases/TouchXPRTWebXPRT/0213/prweb10474803.htm

We have also released the TouchXPRT 2013 source to the community. The instructions for building the benchmark are the same as for building the community preview. Remember that community members have access to the source, but it is not available to the general public.

As WebXPRT 2013 moves from being a community preview to a public release, people may have concerns about privacy. While anyone using WebXPRT 2013 agrees to share their results, the database does not store any identifying information.

Remember, the HDXPRT comment period is still open! Please send your comments in. If you’ve not read the RFC yet, you can find it at http://www.principledtechnologies.com/hdxprt/forum/hdxprt2013RFC.php.

Eric

Comment on this post in the forums

Check out the other XPRTs:

Forgot your password?