Category: Benchmarking

Staying Awake

on February 19, 2015

Here in the Eastern US, we’re in a deep freeze. Wherever you are, I hope you’re comfortable and safe.

Tomorrow, we’ll be releasing a patch to MobileXPRT. This patch fixes a problem first reported in a Tom’s Hardware review of the LG G3. The reviewer was not able to get MobileXPRT to run to completion on that device. We looked at the problem and found that the LG G3 was going to sleep during the performance test, and that when the device woke up, MobileXPRT caught an exception and crashed. This problem appeared to be specific to the LG G3, but could occur on other devices as well.

When we changed MobileXPRT to keep the screen active during the run of the app, the benchmark ran on LG G3 without any issues. We’ve tested this fix on our entire test bed and found that it works well. As always, the results remain comparable with previous versions.

We’ve made the same change in BatteryXPRT and are testing it now. Testing BatteryXPRT takes more time than testing MobileXPRT, but we hope to release the patch soon.

In other news, Bill is going to Mobile World Congress in Barcelona, March 2-4, and he would love to meet you. If you are also planning to go, let us know.

Eric

Posted in Android, Battery life, BatteryXPRT 2014 for Android, Mobile World Congress, MobileXPRT |

A guest blog from Wilson Cheng at Acer Inc.

By Wilson Cheng

on February 12, 2015

Two years ago, I got a job to evaluate cross-platform benchmarks for comparing Windows and Android.

Usually, Web-based benchmarks focus on only Java-script and HTML5 functionality. I found WebXPRT is easy to use and includes the tasks that you do every day. It reliably tests the user experience and gives a simple score for comparison. I suggested to our manager that we use it to evaluate our platform and he agreed.

Recently, Chromebooks have become very popular. Testing the battery life of Chromebooks has been difficult because you have needed to use the power_LoadTest image from the Chromium project or the test image from the Chrome OS Partners Web site to do power load testing. Without reimaging the system, you couldn’t test battery life. Because CrXPRT is designed to test battery life as well as performance on ChromeOS, this extra step is no longer needed. We can run CrXPRT without updating the test image, which is very useful in our battery life testing.

The advantage of XPRT benchmarks is that they are easy to use and give real-world results. The disadvantage is that they are not yet popular on review sites.

Posted in Android, BenchmarkXPRT, Chrome OS, Chromebooks, CrXPRT, Google Chrome, WebXPRT, What makes a good benchmark? |

An updated CrXPRT 2015 build is available

By Eric Hale

on February 10, 2015

Today we’re releasing a new build of CrXPRT 2015 (v1.0.2) in the Chrome Web Store. The updates included in this build correct the handling of outlier scores for individual workloads, check the duration of each battery-life test iteration, and introduce a check to ensure that the system is not plugged in during battery-life testing. The tests have not changed, and the method of calculating the overall score and battery-life score is the same, so the main scores are comparable with previous CrXPRT scores.

We published the update today in the Chrome Web Store as of 3:00 PM EST, but it may take some time for the update to appear on your system. You may have to manually approve the update notice when it appears on your system.

If you have any questions about the update or any other XPRT-related topic, feel free to contact us at BenchmarkXPRTsupport@principledtechnologies.com.

Posted in Battery life, Chrome OS, Chromebooks, CrXPRT, Google Chrome |

Polishing the chrome

By Eric Hale

on February 5, 2015

Early next week, we’ll be releasing an update for CrXPRT that addresses a couple of issues. First, the individual workload scores were not always discarding outliers correctly, which could have had a small effect on those scores. The update handles outliers correctly. Because the overall score is calculated separately from the individual workload scores, this change does not affect the overall score or battery life score, and results from previous builds are still comparable.

The update also checks the duration for each battery life test iteration. This helps detect conditions that could invalidate the test, such as someone manually putting a test system to sleep.

Finally, the update checks between iterations to see if the Chromebook is plugged in during the test. If so, CrXPRT will report an error and stop the test. This can potentially save time that would otherwise be lost on an invalid run.

These are small changes, but they will help make CrXPRT a more robust benchmark.

In other news, we’ve been asking members of the community if they would care to share their experiences and thoughts on the blog. We’re delighted that people are interested, and we’ve already received our first guest post that we’ll be publishing in the next couple of weeks. In the meantime, if you’d like to write a post, or just want to share your thoughts on the XPRTs, please let us know!

Eric

Posted in Battery life, Chrome OS, Chromebooks, Collaborative benchmark development, CrXPRT |

A first look

By Eric Hale

on January 29, 2015

The beta build of WebXPRT 2015 is complete. Although the benchmark is still under development, it looks great!

As we described in the design document, it includes a number of improvements over WebXPRT 2013. There are more, and more demanding, tests. The UI has a cleaner, more modern look, and includes a couple of improvements people have been asking for. There is now descriptive text so that you know what’s happening during the test. Also, a progress indicator in the upper right corner shows you how close the test is to completion. Although the screens aren’t final, here’s a preview of one of the new DNA Sequence Analysis test screens:

One of the great virtues of WebXPRT is that it runs on almost anything. However, that level of compatibility has limited the range of devices WebXPRT can test. WebXPRT 2015 introduces a set of experimental tests. Running these tests is optional and their results are not included in the Overall Score. We hope that folks will use them to evaluate both new technologies and platform-specific technologies. The first Community Preview will include a WebGL-based 3D experimental test. We expect to add new experimental tests over time and we’re hoping community members will contribute tests or ideas for them.

I’m very excited to see the new WebXPRT. We’re testing now, and I hope to be able to let you know when the Community Preview will be available soon.

Eric

Posted in BenchmarkXPRT development community, Community Preview, WebGL, WebXPRT, WebXPRT 2015 |

More than Chromebooks

By Eric Hale

on January 22, 2015

Recently, we got a question from AnandTech asking how hard it would be to get CrXPRT to run on Chrome on Windows.

The short answer is that getting it to run isn’t difficult. However, as we have written about many times in the past, it’s not enough for a benchmark to simply run on a device. The results it produces must be comparable. Even if the benchmark appears to run identically, small differences in timers or how the platform reports its state can have a big impact.

To date, we have been dealing only with Chromebooks of various flavors. However, we’re now testing CrXPRT on a much wider range of devices. The results are generally looking reasonable, although we’re finding some minor issues. For example, the battery information isn’t as granular on some devices as it is on Chromebooks.

As soon as we are sure that CrXPRT is returning reasonable results on the new classes of devices, you’ll be the first to know!

In other news, we’re planning to remove TouchXPRT 2013 from the Windows Store on February 16. We wanted to have a period of overlap with TouchXPRT 2014 to allow labs time to transition. It’s been over 6 months, and we feel this is a good time. TouchXPRT 2013 will remain available in the members’ area of the BenchmarkXPRT.com Web site.

Eric

Posted in BenchmarkXPRT development community, Chrome OS, Collaborative benchmark development, CrXPRT, TouchXPRT, What makes a good benchmark?, Windows 8, Windows 8.1 |

Category: Benchmarking

Staying Awake

A guest blog from Wilson Cheng at Acer Inc.

An updated CrXPRT 2015 build is available

Polishing the chrome

A first look

More than Chromebooks

Check out the other XPRTs: