Category: Benchmarking

A new model

on November 20, 2014

Today we released a new community preview for CrXPRT (CP2). It contains several enhancements and UI changes. It also fixes the problem we discussed in last week’s blog, so CP2 will now correctly execute a rundown test on a system using Chinese language settings.

The most exciting enhancement is that CP2 will let you test battery life without having to put the device into developer mode. The new Battery Status API, which became available with Chrome 38, makes this possible.

Another enhancement is that results submission is now integrated into the benchmark UI. CrXPRT follows the WebXPRT model, with you having opt-in control over whether the results are published on the PT Web site. However, CrXPRT goes further by allowing you the option to upload more extensive disclosure information, and the option to download results as a text file.

These and other enhancements are covered in more detail in the CrXPRT forum and the user manual. Because this is a community preview, you have to be a community member to download it. However, joining is very easy.

We hope you like the new features in CrXPRT. Let us know what you think!

Eric

Discuss CrXPRT CP2 in the forums

Posted in Chrome OS, Collaborative benchmark development, Community Preview, CrXPRT |

News about MobileXPRT 2013 and WebXPRT 2015

By Eric Hale

on November 14, 2014

Today, we’re releasing a new build (v92) of MobileXPRT 2013 at MobileXPRT.com and the Google Play store. This build addresses issues encountered when testing MobileXPRT on Android 5 devices. The tests have not changed, so new scores are comparable with previous MobileXPRT 2013 scores.

Click here to download the new MobileXPRT build directly from our site.

Alternatively, you can download the MobileXPRT 2013 APKs directly. The APKs are small (17.9 MB total) and allow you to download the test content during installation. For users who have trouble accessing the Google Play store, these APKs may make it easier to download the benchmark.

Download the MobileXPRT APK here.
Download the MobileXPRT UX Tests APK here.

Also, today we’re publishing the WebXPRT 2015 Design Overview document for community members. You can find the document on the WebXPRT tab in the Members’ Area. We look forward to your feedback!

If you have any questions or concerns about these or any other XPRT-related topics, feel free to contact us at BenchmarkXPRTsupport@principledtechnologies.com.

Comment on this post in the forums

Posted in Android, Collaborative benchmark development, MobileXPRT, WebXPRT, WebXPRT 2015 |

Upcoming experiments

By Eric Hale

on November 6, 2014

Next week, we’ll be releasing the design overview for WebXPRT 2015. WebXPRT 2013 has been an enormous success, having been run tens of thousands of times.

One of the big improvements we are considering for WebXPRT 2015 is adding experimental tests. A big reason for WebXPRT’s success is that it runs on almost every Web-enabled device. We consider it essential to preserve this broad compatibility. However, there are interesting Web technologies that simply are not available on all devices.

Our proposal is to add experimental tests to WebXPRT. These tests would be optional and would not be included in the Overall score, so WebXPRT would still be able to compare the performance of widely different devices. We are looking at technologies such as Web Workers, WebGL, and pre-compiled JavaScript (asm.js).

In addition to adding experimental tests, we are looking at ways to improve the UI, add automation, add new tests, update old tests, and more!

If you are a community member, you’ll get a notice when the overview is available. We will definitely want to know what you think! If you are not a member, it’s a great time to join.

If you have any thoughts on these ideas, or have ideas of your own, please let us know!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Performance benchmarking, WebXPRT, WebXPRT 2015, What makes a good benchmark? |

Getting the most from the XPRTs

By Eric Hale

on October 30, 2014

BatteryXPRT can measure battery life three ways: in Airplane mode, Wireless connection mode, and Cellular connection mode.

A couple of weeks ago, Tom’s Hardware ran a review of the NVIDIA Shield Tablet that shows the value of being able to compare different ways of using your device. The review gave results for two of the three modes, Airplane mode and Wireless mode, for the devices under test. While all devices had lower battery life in Wireless mode, the NVIDIA Shield Tablet showed a much larger difference between the two modes than other devices.

The review offers some technical reasons why this might be so. However, the review also includes a sentence that goes to the heart of our mission to provide easy-to-use tools that reflect real-world usage and can be used in a variety of ways. As the reviewer at Tom’s Hardware noted: “I’ve also noticed what I would consider excessive power drain during standby with Wi-Fi left on, which subjectively corroborates these results.” That is what we like to hear!

We are always looking for ways to make our tools more versatile and useful. If you have ideas, please let us know!

Eric

Comment on this post in the forums

Posted in Android, Battery life, BatteryXPRT 2014 for Android, Benchmarking, BenchmarkXPRT |

CrXPRT is here!

By Eric Hale

on October 15, 2014

Today we are releasing the CrXPRT 2014 Community Preview (CP1). As mentioned in a previous post, CrXPRT contains performance and battery life tests. The performance suite includes five scenarios utilizing Web browsing and JavaScript workloads, plus Portable Native Client (PNaCl) and WebGL-based scenarios. The battery life test incorporates all of the performance workloads and adds video playback, audio playback, and HTML5 gaming scenarios.

The battery life test in CP1 builds on the lessons we learned from developing BatteryXPRT 2014 for Android. In fact, we’ve been able to improve on the testing time. BatteryXPRT 2014 requires 5.5 hours to estimate battery life; CP1 can estimate battery life in only 3.5 hours. The battery test in CP1 still requires the device be put in developer mode, so we’re investigating the new Chrome OS battery status APIs. We hope these will make it possible to remove this restriction in a future release.

The estimates for battery life are generally pretty accurate. However, we have seen runs where the battery life results were much higher than expected. We are continuing to investigate this. If you see an anomalous result, please let us know. It is worth noting that the performance scores have been very consistent.

Because this is a community preview, you have to be a community member to download it. However, joining is very easy.

Check out the new CrXPRT, and let us know what you think!

Eric

Comment on this post in the forums

Posted in Battery life, BenchmarkXPRT development community, Chrome OS, Community Preview, CrXPRT, Performance benchmarking |

News from the factory floor

By Eric Hale

on October 9, 2014

As we mentioned last week, we have BatteryXPRT and MobileXPRT news:

Today, we’re releasing a new build of MobileXPRT 2013 at MobileXPRT.com and the Google Play store. This build addresses issues we saw when testing MobileXPRT on the beta build of Android L and the experimental ART runtime. The tests have not changed, and the scores are comparable with previous MobileXPRT 2013 scores.

Also, the BatteryXPRT 2014 for Android APKs are now available at BatteryXPRT.com. Up to now, only the full installer, including the content for the tests, was available on our Web site. The APKs are much smaller and allow you to download the test content during installation. For users who have trouble accessing the Google Play store, these APKs may make it easier to download the benchmark.

If you have any questions or concerns, please don’t hesitate to contact us at BenchmarkXPRTsupport@principledtechnologies.com.

Don’t forget: we’re releasing the community preview of CrXPRT next week!

Eric

Comment on this post in the forums

Posted in Android, BatteryXPRT 2014 for Android, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Chrome OS, Community Preview, CrXPRT, MobileXPRT |

Category: Benchmarking

A new model

News about MobileXPRT 2013 and WebXPRT 2015

Upcoming experiments

Getting the most from the XPRTs

CrXPRT is here!

News from the factory floor

Check out the other XPRTs: