Tag Archives: Android

A BatteryXPRT bug fix is on the way

on October 18, 2018

Some time ago, we started to see unusual BatteryXPRT battery life estimates and high variance on some devices when running tests at the default length of 5.25 hours (seven 45-minute iterations). We suspected that the problem resulted from changes in how new OS versions report battery life on certain devices (e.g., charging past a reported level of 100 percent). In addition, the progress of battery technology in general means that the average phone battery lasts much longer than it did a few years ago. Together, these factors sometimes led to BatteryXPRT runs where the OS reported little to no battery decrease during the first few iterations of a test. We concluded that 5.25 hours wasn’t long enough to produce an accurate battery life estimate.

After extensive experimentation and testing, we’ve decided to release a new build that increases the default BatteryXPRT test length from 5.25 hours (seven iterations) to 45 hours (60 iterations) to allow enough time for a full rundown on most phones. Based on our testing, we consider full rundown tests to be the most accurate and will use those exclusively in our Spotlight testing and elsewhere. Testers will still have the option of choosing shorter test durations, but BatteryXPRT will flag the results with a qualifier that recommends performing a full rundown.

We plan to release the updated build by the end of next week and update BatteryXPRT documentation to reflect the changes. We have not changed any of the workloads and both performance results and full-rundown battery life estimates will be comparable to results from earlier builds.

BatteryXPRT continues to be a useful tool for gauging the performance and expected battery life of Android devices while simulating real-world tasks. If you have any questions about BatteryXPRT, please feel free to ask!

Justin

Posted in Android, Battery life, BatteryXPRT 2014 for Android, Benchmarking, BenchmarkXPRT, Mobile devices, Performance benchmarking | Also tagged battery life, BatteryXPRT, mobile devices, XPRT Spotlight

Updates on HDXPRT 4 and MobileXPRT 3

By Justin Greene

on October 11, 2018

There’s a lot going on with the XPRTs, so we want to offer a quick update.

On the HDXPRT 4 front, we’re currently testing community preview candidate builds across a variety of laptops and desktops. Testing is going well, but as is often the case prior to a release, we’re still tweaking the code as necessary when we run into bugs. We’re excited about HDXPRT 4 and look forward to the community seeing how much faster and easier to use it is than previous versions. You can read more about what’s to come in HDXPRT 4 here.

On the MobileXPRT 3 front, proof-of-concept testing for the new and updated workloads went well, and we’re now working to implement the new UI. Below, you can see a mockup of the new MobileXPRT 3 start screen for phones. The aesthetic is completely different than MobileXPRT 2015, and is in line with the clean, bright look we used for WebXPRT 3 and HDXPRT 4. We’ve made it easy to select and deselect individual workloads by tapping the workload name (deselected workloads are grayed out), and we’ve consolidated common menu items into an Android-style taskbar at the bottom of the screen. Please note that this is an early view and some aspects of the screen will change. For instance, we’re certain that the final receipt-scanning workload won’t be called “Optical character recognition.”

We’ll share more information about HDXPRT 4 and MobileXPRT 3 in the coming weeks. If you have any questions about HDXPRT or MobileXPRT, or would like to share your ideas, please get in touch!

Justin

Posted in Android, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, HDXPRT, HDXPRT 4, HDXPRT development process, HDXPRT release cycle, MobileXPRT, MobileXPRT 2015, MobileXPRT 3, Performance benchmarking, WebXPRT, WebXPRT 3 | Also tagged Community, HDXPRT, laptops, mobile devices, MobileXPRT, phones

Notes from the lab: choosing a calibration system for MobileXPRT 3

By Justin Greene

on October 4, 2018

Last week, we shared some details about what to expect in MobileXPRT 3. This week, we want to provide some insight into one part of the MobileXPRT development process, choosing a calibration system.

First, some background. For each of the benchmarks in the XPRT family, we select a calibration system using criteria we’ll explain below. This system serves as a reference point, and we use it to calculate scores that will help users understand a single benchmark result. The calibration system for MobileXPRT 2015 is the Motorola DROID RAZR M. We structured our calculation process so that the mean performance score from repeated MobileXPRT 2015 runs on that device is 100. A device that completes the same workloads 20 percent faster than the DROID RAZR M would have a performance score of 120, and one that performs the test 20 percent more slowly would have a score of 80. (You can find a more in-depth explanation of MobileXPRT score calculations in the Exploring MobileXPRT 2015 white paper.)

When selecting a calibration device, we are looking for a relevant reference point in today’s market. The device should be neither too slow to handle modern workloads nor so fast that it outscores most devices on the market. It should represent a level of performance that is close to what the majority of consumers experience, and one that will continue to be relevant for some time. This approach helps to build context for the meaning of the benchmark’s overall score. Without that context, testers can’t tell whether a score is fast or slow just by looking at the raw number. When compared to a well-known standard such as the calibration device, however, the score has more informative value.

To determine a suitable calibration device for MobileXPRT 3, we started by researching the most popular Android phones by market share around the world. It soon became clear that in many major markets, the Samsung Galaxy S8 ranked first or second, or at least appeared in the top five. As last year’s first Samsung flagship, the S8 is no longer on the cutting edge, but it has specs that many current mid-range phones are deploying, and the hardware should remain relevant for a couple of years.

For all of these reasons, we made the Samsung Galaxy S8 the calibration device for MobileXPRT 3. The model in our lab has a Qualcomm Snapdragon 835 SoC, 4 GB of RAM, and runs Android 7.0 (Nougat). We think it has the balance we’re looking for.

If you have any questions or concerns about MobileXPRT 3, calibration devices, or score calculations, please let us know. We look forward to sharing more information about MobileXPRT 3 as we get closer to the community preview.

Justin

Posted in Benchmark metrics, BenchmarkXPRT, BenchmarkXPRT development community, Mobile devices, MobileXPRT, MobileXPRT 3, Performance benchmarking, Performance testing on tablets, What makes a good benchmark? | Also tagged benchmarks, BenchmarkXPRT, Calibration, Droid, Galaxy, mobile, MobileXPRT, Motorola, Performance, phones, Samsung

Planning the next version of MobileXPRT

By Justin Greene

on July 26, 2018

We’re in the early planning stages for the next version of MobileXPRT, and invite you to send us any suggestions you may have. What do you like or not like about MobileXPRT? What features would you like to see in a new version?

When we begin work on a new version of any XPRT, one of the first steps we take is to assess the benchmark’s workloads to determine whether they will provide value during the years ahead. This step almost always involves updating test content such as photos and videos to more contemporary file resolutions and sizes, and it can also involve removing workloads or adding completely new scenarios. MobileXPRT currently includes five performance scenarios (Apply Photo Effects, Create Photo Collages, Create Slideshow, Encrypt Personal Content, and Detect Faces to Organize Photos). Should we stick with these five or investigate other use cases? What do you think?

As we did with WebXPRT 3 and the upcoming HDXPRT 4, we’re also planning to update the MobileXPRT UI to improve the look of the benchmark and make it easier to use.

Crucially, we’ll also build the app using the most current Android Studio SDK. Android development has changed significantly since we released MobileXPRT 2015 and apps must now conform to stricter standards that require explicit user permission for many tasks. Navigating these changes shouldn’t be too difficult, but it’s always possible that we’ll encounter unforeseen challenges at some point during the process.

Do you have suggestions for test scenarios that we should consider for MobileXPRT? Are there existing features we should remove? Are there elements of the UI that you find especially useful or have ideas for improving? Please let us know. We want to hear from you and make sure that MobileXPRT continues to meet your needs.

Justin

Posted in Android, Benchmark metrics, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Mobile devices, MobileXPRT, MobileXPRT 2015, MobileXPRT 3, Phones, Tablets | Also tagged benchmark, benchmarks, mobile, MobileXPRT, MobileXPRT 2015, MobileXPRT 3, phones, photos, tablets

Keeping up with the latest Android news

By Justin Greene

on September 7, 2017

Ars Technica recently published a deep-dive review of Android 8.0 (Oreo) that contains several interesting tidbits about what the author called “Android’s biggest re-architecture, ever.” After reading the details, it’s hard to argue with that assessment.

The article’s thorough analysis includes a list of the changes Oreo is bringing to the UI, notification settings, locations service settings, and more. In addition to the types of updates that we usually see, a few key points stand out.

Project Treble, a complete reworking of Android’s foundational structure intended to increase the speed and efficiency of update delivery
A serious commitment to eliminating silent background services, giving users more control over their phone’s resources, and potentially enabling significant gains in battery life
Increased machine learning/neural network integration for text selection and recognition
A potential neural network API that allows third-party plugins
Android Go, a scaled-down version of Android tuned for budget phones in developing markets

There’s too much information about each of the points to discuss here, but I encourage anyone interested in Android development to check out the article. Just be warned that when they say “thorough,” they mean it, so it’s not exactly a quick read.

Right now, Oreo is available on only the Google Pixel and Pixel XL phones, and will not likely be available to most users until sometime next year. Even though widespread adoption is a way off, the sheer scale of the expected changes requires us to adopt a long-term development perspective.

We’ll continue to track developments in the Android world and keep the community informed about any impact that those changes may have on MobileXPRT and BatteryXPRT. If you have any questions or suggestions for future XPRT/Android applications, let us know!

Justin

Posted in Android, Android O, Battery life, Google, Machine learning, Mobile devices, MobileXPRT, Open Source, Phones | Also tagged BatteryXPRT, MobileXPRT, Oreo

Tag Archives: Android

A BatteryXPRT bug fix is on the way

Updates on HDXPRT 4 and MobileXPRT 3

Notes from the lab: choosing a calibration system for MobileXPRT 3

Planning the next version of MobileXPRT

Keeping up with the latest Android news

Check out the other XPRTs: