Category: MobileXPRT

It’s almost here!

on August 13, 2015

This Friday, we will be releasing the community preview of MobileXPRT 2015. BenchmarkXPRT Development Community members will be able to download the installation APK from the MobileXPRT tab in the Members’ Area. We are happy to say that, although we thought that MobileXPRT 2015 would only run on Android 5.0 and above, we have been able to extend its compatibility back to Android 4.4 (KitKat).

As we’ve said, the new MobileXPRT is a 64-bit application. Although it will work on both 32-bit and 64-bit hardware, the code will automatically select 64-bit when available. If you want to compare 32-bit performance to 64-bit on a single device, MobileXPRT 2013 will still be available. It’s not a problem to have both versions of MobileXPRT installed at the same time, and results from the two versions of the benchmark are comparable. On the same note, remember that the UX tests are no longer available in MobileXPRT 2015, but they are available as part of MobileXPRT 2013 if you need them.

As always, the results from the community preview will be comparable to the general release.

We hope you enjoy the new MobileXPRT, and we look forward to seeing your results!

Eric

An update on MobileXPRT 2015

By Eric Hale

on July 23, 2015

As I mentioned last week, we’ve been testing MobileXPRT 2015.

We’ve rebuilt the current MobileXPRT as a 64-bit application. This means that MobileXPRT 2015 will only run on Android 5.0 and above. For this reason, we’ll make MobileXPRT 2013 available for testing older versions of the operating system. Because the workloads haven’t changed, you’ll be able to compare results from MobileXPRT 2013 to those from MobileXPRT 2015.

Another change to MobileXPRT 2015 will be the removal of the UX tests. While these tests were useful on lower-end devices when MobileXPRT 2013 came out, they don’t distinguish modern devices. However, should you need them, these tests will continue to be available as part of MobileXPRT 2013.

There are a few other features that we’ll be talking about over the next few weeks. We’ll be releasing the MobileXPRT 2015 community preview before Android M is released. We’re looking forward to see how Android M performs!

Eric

Posted in Android, Android L, Android M, Collaborative benchmark development, Community Preview, MobileXPRT 2015 |

Get the 411 on WebXPRT 2015

By Eric Hale

on July 16, 2015

We’re publishing the WebXPRT 2015 white paper tomorrow. It contains lots of information about WebXPRT 2015 that you won’t find anywhere else. In addition to explaining what the tests do, the test descriptions detail the particular aspects of Web browsing performance that the test stresses.

The white paper also contains a list of the third-party libraries that WebXPRT uses, what those libraries do, and where you can find them. It also lists the HTML 5 capabilities that WebXPRT 2015 requires. And, as in past XPRT white papers, we include a spreadsheet showing how the benchmark calculates its results. There’s a lot of information, so check it out!

Meanwhile, we’re hard at work on MobileXPRT 2015. The Android M Developer Preview 2 hasn’t presented any new problems, so things are looking good. I expect to have more details about the upcoming MobileXPRT 2015 community preview soon!

Eric

Posted in Android, Android M, MobileXPRT, MobileXPRT 2015, WebXPRT 2015, White papers |

Bit by bit

By Eric Hale

on June 11, 2015

We’ve been working to internationalize the XPRTS. Our initial attempts have focused on China.

Both BatteryXPRT and WebXPRT have Chinese UI options. We expect to have a version of MobileXPRT with a Chinese UI option available in a couple of weeks.

We’ve also been working to make the benchmarks more accessible in China. WebXPRT has a mirror host site in Singapore. We’re also getting the XPRTs into Chinese app stores. MobileXPRT is available in two Chinese app stores: Xiaomi (http://app.mi.com/detail/90862) and Zhushou 360 (http://zhushou.360.cn/detail/index/soft_id/2984653). We aim to have BatteryXPRT and the Chinese version of MobileXPRT available in those stores as quickly as possible.

Obviously, we will continue to work to improve our localization. This is an area where we can use the help of the community. If you have the translation skills and want to contribute the strings for a UI for your language, let us know.

Eric

Posted in BenchmarkXPRT, MobileXPRT, WebXPRT |

Rolling with the changes

By Eric Hale

on June 4, 2015

While WebXPRT 2015 has been running fine on earlier beta versions of Windows 10, we have found a problem on some recent versions. Starting with build 10122, the Local Notes test hangs when using the Microsoft Edge browser. (Note: This browser still identifies itself as Spartan in the builds we have seen.) Chrome and Firefox on Windows 10 have successfully run WebXPRT 2015, so the problem appears to be restricted to Edge/Spartan.

Because WebXPRT ran successfully on earlier builds of Windows 10, we are hoping that upcoming builds will resolve the problem. However, we have been investigating the issue in case there is something we can address. The problem is that the encrypted strings that the test is trying to write to LocalStorage are not being written correctly. Non-encrypted strings are being written correctly.

As soon as the problem gets resolved, we’ll let you know.

In other news, we’ve been looking at Android M. There are a lot of interesting changes coming, such as the difference in the way that Android M manages app permissions. We’ve decided to delay releasing the design document for the next version of MobileXPRT so that we can make sure that the design deals with these changes appropriately.

Eric

Posted in Android, Benchmarking, Chrome OS, Collaborative benchmark development, MobileXPRT, WebXPRT, WebXPRT 2015, Windows 10 |

What makes a good benchmark?

By Eric Hale

on May 28, 2015

As we discussed recently, we’re working on the design document for the next version of MobileXPRT, and we’re really interested in any ideas you may have. However, we haven’t talked much about what makes for a good benchmark test.

The things we measure need to be quantifiable. A reviewer can talk about the realism of game play, or the innovative look of a game, and those are valid observations. However, it is difficult to convert those kinds of subjective impressions to numbers.

The things we measure must also be repeatable. For example, the response time for an online service may depend on the time of day, number of people using the service at the time, network load, and other factors that change over time. You can measure the responsiveness of such services, but doing so requires repeating the test enough times under enough different circumstances to get a representative sample.

The possible things we can measure go beyond the speed of the device to include things such as battery life and compatibility with standards, and even fidelity or quality such as with photos or video. BatteryXPRT and CrXPRT test battery life, while the HTML5 tests in WebXPRT are among those that test compatibility. We are currently looking into quality metrics for possible future tools.

I hope this has given you some ideas. If so, let us know!

Eric

Posted in Benchmarking, Benchmarks in general, BenchmarkXPRT development community, MobileXPRT |

Category: MobileXPRT

It’s almost here!

An update on MobileXPRT 2015

Get the 411 on WebXPRT 2015

Bit by bit

Rolling with the changes

What makes a good benchmark?

Check out the other XPRTs: