Category: Collaborative benchmark development

Finally!

on March 26, 2015

We’re releasing the community preview of WebXPRT 2015 tomorrow. We’re very excited that it’s finally here. In the past few weeks, we’ve discussed some of the new features in WebXPRT 2015, such as test automation, its new and improved tests, and its Chinese UI. We think you’re really going to enjoy the new WebXPRT.

The design document (login required) specifies that WebXPRT will contain an experimental workload. That workload is not in the community preview, although we plan for it to be in the general release. However, because any experimental workloads are not included in the overall score, this will not affect any results you generate.

We’re also investigating the use of the JavaScript navigator object to improve system disclosure, but we are still determining if we can get reliable enough information to display. So this information is not displayed in the community preview.

As with all the BenchmarkXPRT community previews, we’re not putting any publication restrictions on this preview release. Test at will, and publish your findings. We guarantee the results for the community preview will be comparable to results from the general release.

If you’re not a community member, join us and check out the new WebXPRT.

Eric

Posted in Automation, BenchmarkXPRT development community, Chinese, Collaborative benchmark development, Community Preview, WebXPRT, WebXPRT 2015 |

More power, more control

By Eric Hale

on March 19, 2015

As I said last week, the community preview for WebXPRT 2015 is coming up soon. One of the changes that will be exciting to anyone who does a lot of testing is that we made it simpler to automate WebXPRT tests.

WebXPRT 2015 will let you automatically select any set of tests you want to run. However, as always, you must run the entire suite of tests to get an overall score. Although the community preview will not include any experimental tests, the automation includes control for those future tests as well.

You may choose from several output formats: HTML table, XML, and CSV, or you can download the results as a text file.

Using the automation is simple: you just append the desired test parameters to the end of the URL. The format allows you to mix and match a lot of options, while still being very concise. The details will be in the release notes.

As people who test a lot of devices, we are very excited about this new capability.

Eric

Posted in Automation, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Future of performance evaluation, Performance benchmarking, WebXPRT, What makes a good benchmark? |

Polishing the chrome

By Eric Hale

on February 5, 2015

Early next week, we’ll be releasing an update for CrXPRT that addresses a couple of issues. First, the individual workload scores were not always discarding outliers correctly, which could have had a small effect on those scores. The update handles outliers correctly. Because the overall score is calculated separately from the individual workload scores, this change does not affect the overall score or battery life score, and results from previous builds are still comparable.

The update also checks the duration for each battery life test iteration. This helps detect conditions that could invalidate the test, such as someone manually putting a test system to sleep.

Finally, the update checks between iterations to see if the Chromebook is plugged in during the test. If so, CrXPRT will report an error and stop the test. This can potentially save time that would otherwise be lost on an invalid run.

These are small changes, but they will help make CrXPRT a more robust benchmark.

In other news, we’ve been asking members of the community if they would care to share their experiences and thoughts on the blog. We’re delighted that people are interested, and we’ve already received our first guest post that we’ll be publishing in the next couple of weeks. In the meantime, if you’d like to write a post, or just want to share your thoughts on the XPRTs, please let us know!

Eric

Posted in Battery life, Chrome OS, Chromebooks, Collaborative benchmark development, CrXPRT |

More than Chromebooks

By Eric Hale

on January 22, 2015

Recently, we got a question from AnandTech asking how hard it would be to get CrXPRT to run on Chrome on Windows.

The short answer is that getting it to run isn’t difficult. However, as we have written about many times in the past, it’s not enough for a benchmark to simply run on a device. The results it produces must be comparable. Even if the benchmark appears to run identically, small differences in timers or how the platform reports its state can have a big impact.

To date, we have been dealing only with Chromebooks of various flavors. However, we’re now testing CrXPRT on a much wider range of devices. The results are generally looking reasonable, although we’re finding some minor issues. For example, the battery information isn’t as granular on some devices as it is on Chromebooks.

As soon as we are sure that CrXPRT is returning reasonable results on the new classes of devices, you’ll be the first to know!

In other news, we’re planning to remove TouchXPRT 2013 from the Windows Store on February 16. We wanted to have a period of overlap with TouchXPRT 2014 to allow labs time to transition. It’s been over 6 months, and we feel this is a good time. TouchXPRT 2013 will remain available in the members’ area of the BenchmarkXPRT.com Web site.

Eric

Posted in BenchmarkXPRT development community, Chrome OS, Collaborative benchmark development, CrXPRT, TouchXPRT, What makes a good benchmark?, Windows 8, Windows 8.1 |

CrXPRT 2015 is here!

By Eric Hale

on December 18, 2014

Today, we’re releasing CrXPRT 2015, our benchmark for evaluating the performance and battery life of Chrome OS devices. The BenchmarkXPRT Development Community has been using community previews for several weeks, but now that we’ve released the benchmark, anyone can download CrXPRT and publish results.

Download CrXPRT 2015 from the Chrome Web Store

You can read more about CrXPRT in the CrXPRT 2015 User Manual located here. The User Manual also explains how to configure your device for testing and discusses the options that are available to you.

After trying out CrXPRT, please send your comments to BenchmarkXPRTsupport@principledtechnologies.com.

Comment on this post in the forums

Posted in Chrome OS, Chromebooks, Collaborative benchmark development, CrXPRT |

A very good question

By Eric Hale

on December 4, 2014

A couple of weeks ago, we released the second community preview of CrXPRT (CP2). One of the changes was to make CrXPRT handle its test results more like WebXPRT. It now automatically uploads some test data to the PT servers, so that we can use it to improve future versions of the benchmark. Like WebXPRT, no personally identifying information is collected, and the results are never made public without your permission.

This week we got a question about the new results collection in CP2. The member wanted to know if the uploading of the results affected the battery life score at all. It does not. The results collection happens after the test. If you’re doing a battery rundown test, the results aren’t uploaded until the device boots the next time. I’m sorry we didn’t make that clearer in the documentation.

When we release a community preview, we commit that the results from that preview will be comparable to any subsequent previews and to the general release. In the case of CP2, we compared the results from CP2 to those from CP1 to make sure that the results stayed comparable. We take this commitment very seriously. When you test with a community preview, you can be sure that the results will still be good when the general release comes out.

In other news, we released a patch for BatteryXPRT yesterday. As we explained in the last blog post, this new version of BatteryXPRT is much more stable on devices running Android 5.

Have any other good questions? Send them our way.

Eric

Comment on this post in the forums

Posted in Android, BatteryXPRT 2014 for Android, Chrome OS, Collaborative benchmark development, Community Preview, CrXPRT, WebXPRT |

Category: Collaborative benchmark development

Finally!

More power, more control

Polishing the chrome

More than Chromebooks

CrXPRT 2015 is here!

A very good question

Check out the other XPRTs: