BenchmarkXPRT Blog banner

Category: Collaborative benchmark development

More than Chromebooks

Recently, we got a question from AnandTech asking how hard it would be to get CrXPRT to run on Chrome on Windows.

The short answer is that getting it to run isn’t difficult. However, as we have written about many times in the past, it’s not enough for a benchmark to simply run on a device. The results it produces must be comparable. Even if the benchmark appears to run identically, small differences in timers or how the platform reports its state can have a big impact.

To date, we have been dealing only with Chromebooks of various flavors. However, we’re now testing CrXPRT on a much wider range of devices. The results are generally looking reasonable, although we’re finding some minor issues. For example, the battery information isn’t as granular on some devices as it is on Chromebooks.

As soon as we are sure that CrXPRT is returning reasonable results on the new classes of devices, you’ll be the first to know!

In other news, we’re planning to remove TouchXPRT 2013 from the Windows Store on February 16. We wanted to have a period of overlap with TouchXPRT 2014 to allow labs time to transition. It’s been over 6 months, and we feel this is a good time. TouchXPRT 2013 will remain available in the members’ area of the BenchmarkXPRT.com Web site.

Eric

CrXPRT 2015 is here!

Today, we’re releasing CrXPRT 2015, our benchmark for evaluating the performance and battery life of Chrome OS devices. The BenchmarkXPRT Development Community has been using community previews for several weeks, but now that we’ve released the benchmark, anyone can download CrXPRT and publish results.

Download CrXPRT 2015 from the Chrome Web Store

You can read more about CrXPRT in the CrXPRT 2015 User Manual located here. The User Manual also explains how to configure your device for testing and discusses the options that are available to you.

After trying out CrXPRT, please send your comments to BenchmarkXPRTsupport@principledtechnologies.com.

Comment on this post in the forums

A very good question

A couple of weeks ago, we released the second community preview of CrXPRT (CP2). One of the changes was to make CrXPRT handle its test results more like WebXPRT. It now automatically uploads some test data to the PT servers, so that we can use it to improve future versions of the benchmark. Like WebXPRT, no personally identifying information is collected, and the results are never made public without your permission.

This week we got a question about the new results collection in CP2. The member wanted to know if the uploading of the results affected the battery life score at all. It does not. The results collection happens after the test. If you’re doing a battery rundown test, the results aren’t uploaded until the device boots the next time. I’m sorry we didn’t make that clearer in the documentation.

When we release a community preview, we commit that the results from that preview will be comparable to any subsequent previews and to the general release. In the case of CP2, we compared the results from CP2 to those from CP1 to make sure that the results stayed comparable. We take this commitment very seriously. When you test with a community preview, you can be sure that the results will still be good when the general release comes out.

In other news, we released a patch for BatteryXPRT yesterday. As we explained in the last blog post, this new version of BatteryXPRT is much more stable on devices running Android 5.

Have any other good questions? Send them our way.

Eric

Comment on this post in the forums

An update for BatteryXPRT 2014 for Android

For those of you in the US, I hope you had a happy Thanksgiving.

A couple of weeks ago, we updated MobileXPRT so that it would run correctly on Android 5 (Android L) devices. Later this week, we will be updating BatteryXPRT as well. The current BatteryXPRT build doesn’t work with Android 5 due to failures of the Create Slideshow workload. This is the same workload failure we fixed with an updated MobileXPRT build.

The new build greatly improves things. We successfully tested the new build on an array of Android 5 and 4.4 devices with no failures. However, we have seen the new build of BatteryXPRT return intermittent errors during the Create Photo Collage and Apply Photo Effects workloads when testing on the Nexus 9.

Because the error on the Nexus 9 is intermittent, it’s possible to get results from some runs. These results are still valid.

We haven’t seen this failure on any device other than the Nexus 9, but cannot guarantee that there aren’t others. If you see any other failures, or have any questions, please contact us.

We’re currently working on a solution for this problem. We will, as always, keep you informed.

Eric

Comment on this post in the forums

A new model

Today we released a new community preview for CrXPRT (CP2). It contains several enhancements and UI changes. It also fixes the problem we discussed in last week’s blog, so CP2 will now correctly execute a rundown test on a system using Chinese language settings.

The most exciting enhancement is that CP2 will let you test battery life without having to put the device into developer mode.  The new Battery Status API, which became available with Chrome 38, makes this possible.

Another enhancement is that results submission is now integrated into the benchmark UI. CrXPRT follows the WebXPRT model, with you having opt-in control over whether the results are published on the PT Web site. However, CrXPRT goes further by allowing you the option to upload more extensive disclosure information, and the option to download results as a text file.

These and other enhancements are covered in more detail in the CrXPRT forum and the user manual. Because this is a community preview, you have to be a community member to download it. However, joining is very easy.

We hope you like the new features in CrXPRT. Let us know what you think!

Eric

Discuss CrXPRT CP2 in the forums

News about MobileXPRT 2013 and WebXPRT 2015

Today, we’re releasing a new build (v92) of MobileXPRT 2013 at MobileXPRT.com and the Google Play store. This build addresses issues encountered when testing MobileXPRT on Android 5 devices. The tests have not changed, so new scores are comparable with previous MobileXPRT 2013 scores.

Click here to download the new MobileXPRT build directly from our site.

Alternatively, you can download the MobileXPRT 2013 APKs directly. The APKs are small (17.9 MB total) and allow you to download the test content during installation. For users who have trouble accessing the Google Play store, these APKs may make it easier to download the benchmark.

Download the MobileXPRT APK here.
Download the MobileXPRT UX Tests APK here.

Also, today we’re publishing the WebXPRT 2015 Design Overview document for community members. You can find the document on the WebXPRT tab in the Members’ Area. We look forward to your feedback!

If you have any questions or concerns about these or any other XPRT-related topics, feel free to contact us at BenchmarkXPRTsupport@principledtechnologies.com.

Comment on this post in the forums

Check out the other XPRTs:

Forgot your password?