Category: WebXPRT 2013 results

How we do it

on September 15, 2016

In the modern world, we’re awash in statistics and it’s interesting how they sometimes contradict each other. In this season, polls are an obvious example. They seldom agree and sometimes, as in the case of Brexit, they can be spectacularly wrong.

The real world is complicated and there are many ways to look at the data. An approach can be valid for certain cases, but less so for others. This is why it’s so important to explain your methods and calculations.

We’ve talked before about the statistics we use in the XPRTs. There are a set of white papers that describe how the tests work and how we perform the calculations. If you’re interested, a great place to start is the WebXPRT 2013 results calculation and confidence interval white paper, which describes the statistics in detail and links to a spreadsheet with a detailed example.

Any methodology can be improved, so if you have any ideas, let us know!

Eric

It’s not the same

By Eric Hale

on May 21, 2015

We sometimes get questions about comparing results from older versions of benchmarks to the current version. Unfortunately, it’s never safe to compare the results from different versions of benchmarks. This principle has been around much longer than the XPRTs. A major update will use different workloads and test data, and will probably be built with updated or different tools.

To avoid confusion, we rescale the results every time we release a new version of an existing benchmark. By making the results significantly different, we hope to reduce the likelihood that results from two different versions will get mixed together.

As an example, we scaled the results from WebXPRT 2015 to be significantly lower than those from WebXPRT 2013. Here are some scores from the published results for WebXPRT 2013 and WebXPRT 2015.

Please note that the results above are not necessarily from the same device configurations, and are meant only to illustrate the difference in results between the two versions of WebXPRT.

If you have any questions, please let us know.

Eric

Posted in BenchmarkXPRT, Collaborative benchmark development, WebXPRT, WebXPRT 2013 results, WebXPRT 2015, What makes a good benchmark? |

Extreme makeover

By Eric Hale

on April 25, 2013

Last week, we unveiled redesigned Web pages for BenchmarkXPRT. We’ve been working on this redesign for a while. We think you’ll find the pages to be a lot sleeker and more attractive. The HDXPRT page, for example, is far less cluttered and easier to navigate. There’s a new white papers page. The members’ area has a new tabbed design that will let you access the member resources for any benchmark form a single page.

We will be redesigning the blog and forums over the next few weeks. Log into the forum or send an e-mail to benchmarkxprtsupport@principledtechnologies.com and tell us what you think about the new design!

As we mentioned in the blog post What a week!, WebXPRT does not collect any personally identifying information. (The WebXPRT data collection page details all the information the benchmark collects.) The benchmark does not attempt to verify that the user agent string is correct under the assumption that the user or browser had some reason for setting it the way it is.

This has caused some people to be confused when, for example, the results for a phone running the stock Android browser say that the phone used Safari. Most modern browsers have the ability to change the user agent string and misidentify themselves, as that version of the Android browser did by default. In fact, you can usually override the browser’s default, should you want to. For example, Google Chrome version 26.0, the version I’m using right now, allows you to choose from multiple versions of IE, Firefox, Chrome, iPhone, iPad, Android, and others. You can even type in a custom string.

So, if you think WebXPRT misidentified your browser, it’s worth checking the user agent string. The instructions for doing this vary by browser, but are usually pretty straightforward. If you’re curious about why browsers offer this feature, you can search for “user agent spoofing” to find explanations of the pros and cons.

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, HDXPRT, Principled Technologies, WebXPRT, WebXPRT 2013 results |

Lies, damned lies, and statistics

By Eric Hale

on April 11, 2013

No one knows who first said “lies, damned lies, and statistics,” but it’s easy to understand why they said it. It’s no surprise that the bestselling statistics book in history is titled How to Lie with Statistics. While the title is facetious, it is certainly true that statistics can be confusing—consider the word “average,” which can refer to the mean, median, or mode. “Mean average,” in turn, can refer to the arithmetic mean, the geometric mean, or the harmonic mean. It’s enough to make a non-statistician’s head spin.

In fact, a number of people have been confused by the confidence interval WebXPRT reports. We believe that the best way to stand behind your results is to be completely open about how you crunch the numbers. To this end, we released the white paper WebXPRT 2013 results calculation and confidence interval this past Monday.

This white paper, which does not require a background in mathematics, explains what the WebXPRT confidence interval is and how it differs from the benchmark variability we sometimes talk about. The paper also gives an overview of the statistical and mathematical techniques WebXPRT uses to translate the raw timing numbers into results.

Because sometimes the devil is in the details, we wanted to augment our overview by showing exactly how WebXPRT calculates results. The white paper is accompanied by a spreadsheet that reproduces the calculations WebXPRT uses. If you are mathematically inclined and would like to suggest improvements to the process, by all means let us know!

Eric

Comment on this post in the forums

Posted in Benchmark metrics, WebXPRT, WebXPRT 2013 results, White papers |

Loose ends

By Eric Hale

on March 21, 2013

As we mentioned last week, the Samsung debuted the Galaxy S4 this past week. It seems to have hit all the expectations – including eye-scrolling. Looking at the reviews, it’s somewhere between the greatest phone ever made and something not quite as good as the iPhone. I’m looking forward to seeing one for myself and hoping someone submits a WebXPRT score for it soon.

We’ll be releasing the HDXPRT 2013 design document tomorrow. As we’ve said, the number one comment has been that HDXPRT 2012 is too big and takes too long to run. We have put a lot of thought into how to trim HDXPRT 2013 and still keep the essential value of the benchmark. We’ve also received some other good feedback that we’re incorporating, such as making installing and running HDXPRT scriptable.

We’ve been doing some investigation during the RFC period, and we’ve encountered problems scripting some of the applications, notably iTunes and PowerDirector. We’re working to overcome these problems but if they prove to be insurmountable, we might have to change the list of applications.

Again, thanks to everyone who commented on the HDXPRT 2013 RFC.

We’ve learned that some WebXPRT users in mainland China are having problems with very slow downloads. The good news is that results from the runs are valid. However, we understand that this is frustrating and are investigating solutions. If you are in China and have experienced slow downloads, please send an e-mail to benchmarkxprtsupport@principledtechnologies.com. We would like your help in evaluating any solutions we come up with.

Speaking of WebXPRT, over the next few weeks we’ll be releasing several white papers that look at WebXPRT results in more depth. The first paper will explain the WebXPRT confidence interval and how it relates to run to run variability, which has been confusing to a number of people. The second paper will look at the effect on the browser on WebXPRT scores. The third paper will look at the influence of the operating system.

Eric

Comment on this post in the forums

Posted in HDXPRT 2012 RFC, HDXPRT 2013, HDXPRT 2013 design document, WebXPRT, WebXPRT 2013 results |

What a week!

By Eric Hale

on February 28, 2013

This has been quite a week for the BenchmarkXPRT family. We kicked off the week by announcing a new benchmark: PhoneXPRT. PhoneXPRT is designed to test small form factor devices, such as smart phones. PhoneXPRT will initially run on Android. For more information, read the press release at http://www.prweb.com/releases/phoneXPRT/pt0213/prweb10461639.htm

There was more to come. Today we formally released TouchXPRT 2013 and WebXPRT 2013. The community and the media have been using these as community previews for weeks now. Now that we’ve released the benchmarks, anyone may freely use them. You can read the press release at http://www.prweb.com/releases/TouchXPRTWebXPRT/0213/prweb10474803.htm

We have also released the TouchXPRT 2013 source to the community. The instructions for building the benchmark are the same as for building the community preview. Remember that community members have access to the source, but it is not available to the general public.

As WebXPRT 2013 moves from being a community preview to a public release, people may have concerns about privacy. While anyone using WebXPRT 2013 agrees to share their results, the database does not store any identifying information.

Remember, the HDXPRT comment period is still open! Please send your comments in. If you’ve not read the RFC yet, you can find it at http://www.principledtechnologies.com/hdxprt/forum/hdxprt2013RFC.php.

Eric

Comment on this post in the forums

Posted in PhoneXPRT, TouchXPRT 2013 release, TouchXPRT 2013 source, WebXPRT 2013 release, WebXPRT 2013 results |

Category: WebXPRT 2013 results

How we do it

It’s not the same

Extreme makeover

Lies, damned lies, and statistics

Loose ends

What a week!

Check out the other XPRTs: