Category: What makes a good benchmark?

More power, more control

on March 19, 2015

As I said last week, the community preview for WebXPRT 2015 is coming up soon. One of the changes that will be exciting to anyone who does a lot of testing is that we made it simpler to automate WebXPRT tests.

WebXPRT 2015 will let you automatically select any set of tests you want to run. However, as always, you must run the entire suite of tests to get an overall score. Although the community preview will not include any experimental tests, the automation includes control for those future tests as well.

You may choose from several output formats: HTML table, XML, and CSV, or you can download the results as a text file.

Using the automation is simple: you just append the desired test parameters to the end of the URL. The format allows you to mix and match a lot of options, while still being very concise. The details will be in the release notes.

As people who test a lot of devices, we are very excited about this new capability.

Eric

Posted in Automation, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Future of performance evaluation, Performance benchmarking, WebXPRT, What makes a good benchmark? |

A guest blog from Wilson Cheng at Acer Inc.

By Wilson Cheng

on February 12, 2015

Two years ago, I got a job to evaluate cross-platform benchmarks for comparing Windows and Android.

Usually, Web-based benchmarks focus on only Java-script and HTML5 functionality. I found WebXPRT is easy to use and includes the tasks that you do every day. It reliably tests the user experience and gives a simple score for comparison. I suggested to our manager that we use it to evaluate our platform and he agreed.

Recently, Chromebooks have become very popular. Testing the battery life of Chromebooks has been difficult because you have needed to use the power_LoadTest image from the Chromium project or the test image from the Chrome OS Partners Web site to do power load testing. Without reimaging the system, you couldn’t test battery life. Because CrXPRT is designed to test battery life as well as performance on ChromeOS, this extra step is no longer needed. We can run CrXPRT without updating the test image, which is very useful in our battery life testing.

The advantage of XPRT benchmarks is that they are easy to use and give real-world results. The disadvantage is that they are not yet popular on review sites.

Posted in Android, BenchmarkXPRT, Chrome OS, Chromebooks, CrXPRT, Google Chrome, WebXPRT, What makes a good benchmark? |

More than Chromebooks

By Eric Hale

on January 22, 2015

Recently, we got a question from AnandTech asking how hard it would be to get CrXPRT to run on Chrome on Windows.

The short answer is that getting it to run isn’t difficult. However, as we have written about many times in the past, it’s not enough for a benchmark to simply run on a device. The results it produces must be comparable. Even if the benchmark appears to run identically, small differences in timers or how the platform reports its state can have a big impact.

To date, we have been dealing only with Chromebooks of various flavors. However, we’re now testing CrXPRT on a much wider range of devices. The results are generally looking reasonable, although we’re finding some minor issues. For example, the battery information isn’t as granular on some devices as it is on Chromebooks.

As soon as we are sure that CrXPRT is returning reasonable results on the new classes of devices, you’ll be the first to know!

In other news, we’re planning to remove TouchXPRT 2013 from the Windows Store on February 16. We wanted to have a period of overlap with TouchXPRT 2014 to allow labs time to transition. It’s been over 6 months, and we feel this is a good time. TouchXPRT 2013 will remain available in the members’ area of the BenchmarkXPRT.com Web site.

Eric

Posted in BenchmarkXPRT development community, Chrome OS, Collaborative benchmark development, CrXPRT, TouchXPRT, What makes a good benchmark?, Windows 8, Windows 8.1 |

So easy a child can do it!

By Eric Hale

on January 15, 2015

Tomorrow we are releasing a new video featuring CrXPRT. This one is set in a school science fair, where “Ellie Smith” explains how she used CrXPRT to help her school decide which Chromebook to buy. We were lucky enough to get a thoroughly professional and charming young actress to play the role of Ellie. (I have a tiny cameo as the guy in the gray sport coat at the back of the room.)

Before we started shooting the video, we asked an actual 10-year-old to install and run CrXPRT. I hate to sound like an old commercial, but it really was so simple that a child could do it!

We also created a faux science report to go with the video. An adult—not a sixth-grader—wrote the report, but the results in it and in the video are real. (You can follow the links in the science report to see the real-world results online.)

When it goes live, you’ll find the video and the report on CrXPRT.com, as well as on YouTube and SlideShare. We hope you’ll enjoy seeing Ellie’s project!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, Chrome OS, Chromebooks, CrXPRT, Google Chrome, Video, What makes a good benchmark? |

Upcoming experiments

By Eric Hale

on November 6, 2014

Next week, we’ll be releasing the design overview for WebXPRT 2015. WebXPRT 2013 has been an enormous success, having been run tens of thousands of times.

One of the big improvements we are considering for WebXPRT 2015 is adding experimental tests. A big reason for WebXPRT’s success is that it runs on almost every Web-enabled device. We consider it essential to preserve this broad compatibility. However, there are interesting Web technologies that simply are not available on all devices.

Our proposal is to add experimental tests to WebXPRT. These tests would be optional and would not be included in the Overall score, so WebXPRT would still be able to compare the performance of widely different devices. We are looking at technologies such as Web Workers, WebGL, and pre-compiled JavaScript (asm.js).

In addition to adding experimental tests, we are looking at ways to improve the UI, add automation, add new tests, update old tests, and more!

If you are a community member, you’ll get a notice when the overview is available. We will definitely want to know what you think! If you are not a member, it’s a great time to join.

If you have any thoughts on these ideas, or have ideas of your own, please let us know!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Performance benchmarking, WebXPRT, WebXPRT 2015, What makes a good benchmark? |

What do you think when you hear “Chromebook”?

By Justin Greene

on September 11, 2014

We’ve been thinking a lot about Chromebooks while doing all of our testing in preparation for the CrXPRT Community Preview. In both the models we’re testing and the ones announced in the press, we’ve seen just how much the Chromebook market is changing. Some folks even claim that Chromebook sales made up 35 percent of US commercial laptop sales in the first half of 2014. What’s even more interesting to us is the wide variety of Chromebooks on the market.

Choosing between Chromebooks these days is becoming more complicated than it used to be. There’s a greater range of hardware choices, and those choices can have a direct impact on performance and battery life. Some Chromebooks offer local storage up to 320 GB, touch screens, and 4G/LTE connectivity. Prices range widely, from $199 to $1,499. Even seemingly comparable systems can perform much differently when put to the test. For instance, we recently tested two Chromebooks separated by only $50 in price, but over 5 hours of estimated battery life!

Whether a consumer’s ultimate purchasing decision is based on price, specs, or a combination of factors, there are few things more valuable to buyers than reliable facts about performance and battery life. Benchmarking is ultimately about gaining useful data for decision making, and that’s why we’re excited about the value that CrXPRT will bring to the Chromebook discussion!

Justin

Comment on this post in the forums

Posted in Chrome OS, CrXPRT, Google Chrome, Performance of computing devices, What makes a good benchmark? |

Category: What makes a good benchmark?

More power, more control

A guest blog from Wilson Cheng at Acer Inc.

More than Chromebooks

So easy a child can do it!

Upcoming experiments

What do you think when you hear “Chromebook”?

Check out the other XPRTs: