Category: What makes a good benchmark?

Upping our game

on August 27, 2015

As we wrote last week, we’re releasing MobileXPRT 2015 to the public tomorrow. Thanks to everyone who helped make the community preview a success!

We’re working on the TouchXPRT 2016 design document and will make it available for the community to review soon.

As you know, we’re always investigating initiatives that could improve our game. We’re continuing to investigate creating experimental tests for future XPRTs. Experimental tests will allow us to maintain broad compatibility for each XPRT tool while providing testers with an opportunity to evaluate cutting-edge technologies.

Another initiative involves looking for new partnerships with people who are not yet part of the community, but could add valuable input to the development process. It’s too soon to say much more about this, but we’re having fruitful conversations and hoping that these partnerships will grow the community even more!

If you have ideas about experimental tests, improving the XPRTs, or expanding the community, please let us know.

Eric

Posted in Android, Collaborative benchmark development, Community Preview, Future of performance evaluation, MobileXPRT, MobileXPRT 2015, TouchXPRT, TouchXPRT 2016, What makes a good benchmark? |

Explaining the BenchmarkXPRT Development Community

By Bill Catchings

on August 6, 2015

Over the last year, I’ve spoken about the XPRT benchmarks with people across America, in China (at IDF Shenzhen), and in Europe (at Mobile World Congress). I regularly found myself having to explain how the BenchmarkXPRT Development Community works. While I was glad to do so, I found myself wishing that the many people I wasn’t able to talk with could also learn about how the community works.

To help make that happen, we’ve developed a simple and engaging video. My (admittedly prejudiced) opinion is that it does a great job of explaining how the community works in less than two minutes. That’s a lot faster than I was able to explain it to folks!

We hope you enjoy the video. And we hope you’ll pass it along to other folks who aren’t already part of the community so they can learn how it works and hopefully be persuaded to join us. Thanks!

Bill

Posted in Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, IDF, What makes a good benchmark? |

Mystery solved

By Eric Hale

on July 2, 2015

As we mentioned a few weeks ago, the WebXPRT Local Notes test would not complete on recent builds of Windows 10 when using the Edge browser. Other browsers complete WebXPRT in recent Windows 10 builds without any problems.

We now know what is causing this behavior. The Local Notes test stores encrypted content in LocalStorage as UTF-16 character encoded Unicode strings. The encrypted content included values that are not considered valid characters in certain use cases. The current Edge implementation treats these characters as undefined and cannot store them. Other browsers may not have had an issue with the characters because of differences in the way they implement LocalStorage.

We’ve been able to work around this by using escape sequences for unsupported Unicode code points. Testing so far has not shown any perceptible change in results, so we believe that we will able to make this change to WebXPRT without compromising the comparability of the results.

Because this issue affects both WebXPRT 2013 and WebXPRT 2015, we’re planning to update both versions. We’ll let you know as soon as they are available.

If you’d like more details about this issue and the fix, please let us know.

Eric

Posted in Performance benchmarking, WebXPRT, WebXPRT 2015, What makes a good benchmark?, Windows 10 |

It’s not the same

By Eric Hale

on May 21, 2015

We sometimes get questions about comparing results from older versions of benchmarks to the current version. Unfortunately, it’s never safe to compare the results from different versions of benchmarks. This principle has been around much longer than the XPRTs. A major update will use different workloads and test data, and will probably be built with updated or different tools.

To avoid confusion, we rescale the results every time we release a new version of an existing benchmark. By making the results significantly different, we hope to reduce the likelihood that results from two different versions will get mixed together.

As an example, we scaled the results from WebXPRT 2015 to be significantly lower than those from WebXPRT 2013. Here are some scores from the published results for WebXPRT 2013 and WebXPRT 2015.

Please note that the results above are not necessarily from the same device configurations, and are meant only to illustrate the difference in results between the two versions of WebXPRT.

If you have any questions, please let us know.

Eric

Posted in BenchmarkXPRT, Collaborative benchmark development, WebXPRT, WebXPRT 2013 results, WebXPRT 2015, What makes a good benchmark? |

More power, more control

By Eric Hale

on March 19, 2015

As I said last week, the community preview for WebXPRT 2015 is coming up soon. One of the changes that will be exciting to anyone who does a lot of testing is that we made it simpler to automate WebXPRT tests.

WebXPRT 2015 will let you automatically select any set of tests you want to run. However, as always, you must run the entire suite of tests to get an overall score. Although the community preview will not include any experimental tests, the automation includes control for those future tests as well.

You may choose from several output formats: HTML table, XML, and CSV, or you can download the results as a text file.

Using the automation is simple: you just append the desired test parameters to the end of the URL. The format allows you to mix and match a lot of options, while still being very concise. The details will be in the release notes.

As people who test a lot of devices, we are very excited about this new capability.

Eric

Posted in Automation, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Future of performance evaluation, Performance benchmarking, WebXPRT, What makes a good benchmark? |

A guest blog from Wilson Cheng at Acer Inc.

By Wilson Cheng

on February 12, 2015

Two years ago, I got a job to evaluate cross-platform benchmarks for comparing Windows and Android.

Usually, Web-based benchmarks focus on only Java-script and HTML5 functionality. I found WebXPRT is easy to use and includes the tasks that you do every day. It reliably tests the user experience and gives a simple score for comparison. I suggested to our manager that we use it to evaluate our platform and he agreed.

Recently, Chromebooks have become very popular. Testing the battery life of Chromebooks has been difficult because you have needed to use the power_LoadTest image from the Chromium project or the test image from the Chrome OS Partners Web site to do power load testing. Without reimaging the system, you couldn’t test battery life. Because CrXPRT is designed to test battery life as well as performance on ChromeOS, this extra step is no longer needed. We can run CrXPRT without updating the test image, which is very useful in our battery life testing.

The advantage of XPRT benchmarks is that they are easy to use and give real-world results. The disadvantage is that they are not yet popular on review sites.

Posted in Android, BenchmarkXPRT, Chrome OS, Chromebooks, CrXPRT, Google Chrome, WebXPRT, What makes a good benchmark? |

Category: What makes a good benchmark?

Upping our game

Explaining the BenchmarkXPRT Development Community

Mystery solved

It’s not the same

More power, more control

A guest blog from Wilson Cheng at Acer Inc.

Check out the other XPRTs: