Category: Benchmarking

An update on MobileXPRT 2015

on July 23, 2015

As I mentioned last week, we’ve been testing MobileXPRT 2015.

We’ve rebuilt the current MobileXPRT as a 64-bit application. This means that MobileXPRT 2015 will only run on Android 5.0 and above. For this reason, we’ll make MobileXPRT 2013 available for testing older versions of the operating system. Because the workloads haven’t changed, you’ll be able to compare results from MobileXPRT 2013 to those from MobileXPRT 2015.

Another change to MobileXPRT 2015 will be the removal of the UX tests. While these tests were useful on lower-end devices when MobileXPRT 2013 came out, they don’t distinguish modern devices. However, should you need them, these tests will continue to be available as part of MobileXPRT 2013.

There are a few other features that we’ll be talking about over the next few weeks. We’ll be releasing the MobileXPRT 2015 community preview before Android M is released. We’re looking forward to see how Android M performs!

Eric

Posted in Android, Android L, Android M, Collaborative benchmark development, Community Preview, MobileXPRT 2015 |

Mystery solved

By Eric Hale

on July 2, 2015

As we mentioned a few weeks ago, the WebXPRT Local Notes test would not complete on recent builds of Windows 10 when using the Edge browser. Other browsers complete WebXPRT in recent Windows 10 builds without any problems.

We now know what is causing this behavior. The Local Notes test stores encrypted content in LocalStorage as UTF-16 character encoded Unicode strings. The encrypted content included values that are not considered valid characters in certain use cases. The current Edge implementation treats these characters as undefined and cannot store them. Other browsers may not have had an issue with the characters because of differences in the way they implement LocalStorage.

We’ve been able to work around this by using escape sequences for unsupported Unicode code points. Testing so far has not shown any perceptible change in results, so we believe that we will able to make this change to WebXPRT without compromising the comparability of the results.

Because this issue affects both WebXPRT 2013 and WebXPRT 2015, we’re planning to update both versions. We’ll let you know as soon as they are available.

If you’d like more details about this issue and the fix, please let us know.

Eric

Posted in Performance benchmarking, WebXPRT, WebXPRT 2015, What makes a good benchmark?, Windows 10 |

Something old, something new

By Eric Hale

on June 25, 2015

Last week, we talked about porting TouchXPRT 2014 to be a Windows 10 universal app. This will let it run on devices running Windows 10 and those running Windows 10 mobile.

We won’t be retiring TouchXPRT 2014 when we release the Windows 10 universal app version. Windows 8 doesn’t support Windows 10 universal apps, but Windows 10 will be able to run Windows 8 applications. This means you’ll also be continue to be able to use TouchXPRT 2014 to test Windows 8 based systems, as well as to compare Windows 8 and Windows 10 performance.

The results from TouchXPRT 2014 and the universal app version of the benchmark won’t be compatible. Even though the test scenarios will be the same, the porting process means that we have to change the APIs the benchmark is using and rebuild the benchmark with different tools.

We’re currently debating changing the way we version the benchmarks. As the number of versions of each benchmark increases, it may make sense to move away from year-based versioning. This will obviously affect what we call the new Windows 10 version of TouchXPRT. If you have any thoughts on this, please let us know!

Eric

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, TouchXPRT, TouchXPRT 2014, TouchXPRT release cycle, Windows 10, Windows 8 |

Hoping for a perfect 10

By Eric Hale

on June 18, 2015

As many of you know by now, the release date for Windows 10 is July 29. As we’ve said before, we are hard at work getting TouchXPRT ready for Windows 10. We’ve succeeded in building TouchXPRT as a universal app, and it’s now running on Windows 10. We haven’t successfully run it on Windows 10 Phone yet, but we’re working on that.

Unfortunately, I can’t share any performance data. The EULA for the current build of Windows 10 (build 10143 as I’m writing this) forbids publishing benchmark results without prior written approval from Microsoft.

We’ll continue testing and refining the porting of TouchXPRT to Windows 10. Our goal is to release it as a universal app to the community in July.

What are your experiences testing Windows 10? We’d love to hear about them!

Eric

Posted in Performance benchmarking, TouchXPRT, TouchXPRT development process, TouchXPRT release cycle, Windows 10 |

Rolling with the changes

By Eric Hale

on June 4, 2015

While WebXPRT 2015 has been running fine on earlier beta versions of Windows 10, we have found a problem on some recent versions. Starting with build 10122, the Local Notes test hangs when using the Microsoft Edge browser. (Note: This browser still identifies itself as Spartan in the builds we have seen.) Chrome and Firefox on Windows 10 have successfully run WebXPRT 2015, so the problem appears to be restricted to Edge/Spartan.

Because WebXPRT ran successfully on earlier builds of Windows 10, we are hoping that upcoming builds will resolve the problem. However, we have been investigating the issue in case there is something we can address. The problem is that the encrypted strings that the test is trying to write to LocalStorage are not being written correctly. Non-encrypted strings are being written correctly.

As soon as the problem gets resolved, we’ll let you know.

In other news, we’ve been looking at Android M. There are a lot of interesting changes coming, such as the difference in the way that Android M manages app permissions. We’ve decided to delay releasing the design document for the next version of MobileXPRT so that we can make sure that the design deals with these changes appropriately.

Eric

Posted in Android, Benchmarking, Chrome OS, Collaborative benchmark development, MobileXPRT, WebXPRT, WebXPRT 2015, Windows 10 |

What makes a good benchmark?

By Eric Hale

on May 28, 2015

As we discussed recently, we’re working on the design document for the next version of MobileXPRT, and we’re really interested in any ideas you may have. However, we haven’t talked much about what makes for a good benchmark test.

The things we measure need to be quantifiable. A reviewer can talk about the realism of game play, or the innovative look of a game, and those are valid observations. However, it is difficult to convert those kinds of subjective impressions to numbers.

The things we measure must also be repeatable. For example, the response time for an online service may depend on the time of day, number of people using the service at the time, network load, and other factors that change over time. You can measure the responsiveness of such services, but doing so requires repeating the test enough times under enough different circumstances to get a representative sample.

The possible things we can measure go beyond the speed of the device to include things such as battery life and compatibility with standards, and even fidelity or quality such as with photos or video. BatteryXPRT and CrXPRT test battery life, while the HTML5 tests in WebXPRT are among those that test compatibility. We are currently looking into quality metrics for possible future tools.

I hope this has given you some ideas. If so, let us know!

Eric

Posted in Benchmarking, Benchmarks in general, BenchmarkXPRT development community, MobileXPRT |

Category: Benchmarking

An update on MobileXPRT 2015

Mystery solved

Something old, something new

Hoping for a perfect 10

Rolling with the changes

What makes a good benchmark?

Check out the other XPRTs: