BenchmarkXPRT Development Community Blog

A happy coincidence

on May 2, 2013

I love new gadgets and even the promise of new ones. Samsung just announced the specs for their upcoming Galaxy Tab 3. Initial reactions to the specs have been somewhat muted to say the least. Basically, they’ve looked at this as only an incremental improvement to the current model. The early rumors of a larger screen and bigger improvements, which turned out to be false, surely contributed to the disappointed reactions.

That being said, some sites claim that the performance of the Galaxy Tab 3 is much higher than the Galaxy Tab 2, particularly regarding graphics. We look forward to verifying these claims ourselves.

Coincidentally, this week we have been playing with an early version of PhoneXPRT (or whatever we end up calling it). So far, things are looking good. We ran it on several devices, including a Samsung Galaxy Tab 2. Like all the XPRT benchmarks, it uses real- world scenarios, which we think result in more useful and accurate results. We’ll talk more about the scenarios in the next few weeks.

It’s a very exciting time in the Android phone and tablet market. I can’t wait to try out subsequent versions of the new benchmark on the latest and greatest Android devices!

Bill

Comment on this post in the forums

Posted in MobileXPRT, Performance testing on tablets, PhoneXPRT, Touch-based devices |

Extreme makeover

By Eric Hale

on April 25, 2013

Last week, we unveiled redesigned Web pages for BenchmarkXPRT. We’ve been working on this redesign for a while. We think you’ll find the pages to be a lot sleeker and more attractive. The HDXPRT page, for example, is far less cluttered and easier to navigate. There’s a new white papers page. The members’ area has a new tabbed design that will let you access the member resources for any benchmark form a single page.

We will be redesigning the blog and forums over the next few weeks. Log into the forum or send an e-mail to benchmarkxprtsupport@principledtechnologies.com and tell us what you think about the new design!

As we mentioned in the blog post What a week!, WebXPRT does not collect any personally identifying information. (The WebXPRT data collection page details all the information the benchmark collects.) The benchmark does not attempt to verify that the user agent string is correct under the assumption that the user or browser had some reason for setting it the way it is.

This has caused some people to be confused when, for example, the results for a phone running the stock Android browser say that the phone used Safari. Most modern browsers have the ability to change the user agent string and misidentify themselves, as that version of the Android browser did by default. In fact, you can usually override the browser’s default, should you want to. For example, Google Chrome version 26.0, the version I’m using right now, allows you to choose from multiple versions of IE, Firefox, Chrome, iPhone, iPad, Android, and others. You can even type in a custom string.

So, if you think WebXPRT misidentified your browser, it’s worth checking the user agent string. The instructions for doing this vary by browser, but are usually pretty straightforward. If you’re curious about why browsers offer this feature, you can search for “user agent spoofing” to find explanations of the pros and cons.

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, HDXPRT, Principled Technologies, WebXPRT, WebXPRT 2013 results |

Three names, two hosts

By Eric Hale

on April 18, 2013

As Bill mentioned a couple of weeks ago in The Name Game, we’ve been considering changing the name of PhoneXPRT. The rationale for this is that the tests in PhoneXPRT are useful for a range of devices, from phones to tablets. We asked for your opinions about the name. After getting your input and talking amongst ourselves, we are considering three possibilities:

Leaving the name unchanged.
Changing the name to MobileXPRT. While this would convey the scope of the benchmark, some people thought the name might be too general.
Changing the name to TouchXPRT for Android. While there is some similarity between the tests in TouchXPRT and the ones we are developing for this benchmark, the two benchmarks would not initially be comparable.

Let us know what you think. We hope to settle the name question soon.

As we mentioned in Loose Ends, some users in mainland China are reporting extremely slow download times when running WebXPRT. We have set up a trial host for WebXPRT in Singapore to see if this improves the situation. Preliminary, US-based tests have seen no significant difference in scores when running from the Singapore host.

If you are in China and want to try running WebXPRT from the new host, you will find it at http://54.251.252.204/webxprt/. Please let us know your experience.

Eric

Comment on this post in the forums

Posted in Android, MobileXPRT, PhoneXPRT, WebXPRT |

Lies, damned lies, and statistics

By Eric Hale

on April 11, 2013

No one knows who first said “lies, damned lies, and statistics,” but it’s easy to understand why they said it. It’s no surprise that the bestselling statistics book in history is titled How to Lie with Statistics. While the title is facetious, it is certainly true that statistics can be confusing—consider the word “average,” which can refer to the mean, median, or mode. “Mean average,” in turn, can refer to the arithmetic mean, the geometric mean, or the harmonic mean. It’s enough to make a non-statistician’s head spin.

In fact, a number of people have been confused by the confidence interval WebXPRT reports. We believe that the best way to stand behind your results is to be completely open about how you crunch the numbers. To this end, we released the white paper WebXPRT 2013 results calculation and confidence interval this past Monday.

This white paper, which does not require a background in mathematics, explains what the WebXPRT confidence interval is and how it differs from the benchmark variability we sometimes talk about. The paper also gives an overview of the statistical and mathematical techniques WebXPRT uses to translate the raw timing numbers into results.

Because sometimes the devil is in the details, we wanted to augment our overview by showing exactly how WebXPRT calculates results. The white paper is accompanied by a spreadsheet that reproduces the calculations WebXPRT uses. If you are mathematically inclined and would like to suggest improvements to the process, by all means let us know!

Eric

Comment on this post in the forums

Posted in Benchmark metrics, WebXPRT, WebXPRT 2013 results, White papers |

Rapid Evolution

By Eric Hale

on April 4, 2013

As Bill mentioned last week, we are considering changing the name of PhoneXPRT, not only because of the interest in using the benchmark scenarios on Android based tablets, but also because the line between phones and tablets has become blurred.

Devices that are too big to be a phone and too small to be a proper tablet are everywhere. PC Magazine says the first true “phablet” – possibly the ugliest portmanteau in the history of technology – was the AT&T EO 440 in 1993. However, it was more 8 years before Samsung had the first really successful phablet, the Galaxy Note. Now, less than 2 years later, there are rumors that Samsung may kill the Note in favor of the Samsung Galaxy Mega.

Obviously, this is one of the fastest evolving areas in tech. This rapid evolution has given us an almost bewildering array for devices, from small phones such as the Sony Xperia Mini, which a child can hold in one hand, to the ASUS Transformer AiO P1801, which has a whopping 18.4” screen! All this speed and diversity obviously pose challenges for the new benchmark, but it makes the work very exciting as well!

We have received comments about the name and we really appreciate those. If you have any thoughts, let us know. We hope to make a decision about whether to change the name soon.

Eric

P.S. I should note: The Transformer AiOP1801 also runs Windows 8, which means it’s a great candidate for TouchXPRT as well.

Comment on this post in the forums

Posted in Android, PhoneXPRT, Touch-based devices, TouchXPRT, Windows 8 |

The name game

By Bill Catchings

on March 29, 2013

There’s a lot going on in the world of the XPRTs. We’re working on HDXPRT 2013, writing white papers, building up our results database, and thinking ahead to the next versions of TouchXPRT and WebXPRT. At the same time, we are giving quite a bit of attention to PhoneXPRT.

As we said in the PhoneXPRT press release last month, PhoneXPRT will use the same kinds of realistic scenarios the other XPRT benchmarks do. These scenarios include tasks that people perform on phones as well as other mobile devices.

We’ve gone back and forth on the seemingly simple question of how to define what a phone is. At one point, defining a phone as a mobile device that you put to your ear seemed to suffice. As phones grow larger and tablets smaller, that doesn’t really hold up.

We also have a decent bit of interest in using the benchmark on Android-based tablets as well as phones. That seems like a good idea to us, as we are all for getting the most out of any benchmark.

One issue, however, is that PhoneXPRT is not a great name for a benchmark that may be commonly used on devices other than phones. So we’ve started thinking about what else we could call it.

As always, we look to the community. Do you think calling the benchmark PhoneXPRT would limit its usefulness for benchmarking tablets? Do you have any ideas for more inclusive names? We really need your feedback here and look forward to getting it. Please send suggestions to benchmarkxprtsupport@principledtechnologies.com or post your comments in the forum. Thanks!

Bill

Comment on this post in the forums

Posted in Android, BenchmarkXPRT, HDXPRT 2013, PhoneXPRT |

A happy coincidence

Extreme makeover

Three names, two hosts

Lies, damned lies, and statistics

Rapid Evolution

The name game

Check out the other XPRTs: