Category: Benchmarking

The MobileXPRT 3 Community Preview is almost here!

on December 13, 2018

We appreciate everyone’s patience and feedback during the MobileXPRT 3 development process. We’re in the process of wrapping up some final testing and expect to release the Community Preview (CP) within the next week.

The first thing testers will notice about the CP is a new UI/UX experience. The aesthetic is completely different than MobileXPRT 2015. We’ve made it easy to select and deselect individual workloads by tapping the workload name, and we’ve consolidated common menu items into an Android-style task bar at the bottom of the screen to improve navigation.

Fundamentally, we rebuilt MobileXPRT with Android Studio SDK 27 to bring it up to date with contemporary Android development standards. While we kept the five workloads from MobileXPRT 2015, we gave one a major overhaul, updated the test content in the remaining four, and added a sixth workload: Optical Character Recognition. You can find more details about these changes in our earlier discussion about MobileXPRT 3 development in the blog.

The screenshots below show the MobileXPRT 3 main screen, how the screen looks when users customize the combination of workloads, the end-of-test results screen, and the archived results page with one test result view expanded. Note that the archived results page also displays the median of all overall test scores completed to date.

We’ll announce the CP in a community message and here on the blog when it goes live. As with all community previews, the MobileXPRT 3 CP will be available only to BenchmarkXPRT Development Community members. If you have any questions about MobileXPRT 3 or joining the community, please let us know. Happy testing!

Justin

Posted in Android, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Mobile devices, MobileXPRT, MobileXPRT 2015, MobileXPRT 3 |

XPRT collaborations: North Carolina State University

By Justin Greene

on December 6, 2018

For those of us who work on the BenchmarkXPRT tools, a core goal is involving new contributors and interested parties in the benchmark development process. Adding voices to the discussion fosters the collaboration and innovation that lead to powerful benchmark tools with lasting relevance.

One vehicle for outreach that we especially enjoy is sponsoring a student project through North Carolina State University. Each semester, the Senior Design Center in the university’s Department of Computer Science partners with external companies and organizations to provide student teams with an opportunity to work on real-world programming projects. If you’ve followed the XPRTs for a while, you may remember previous student projects such as Nebula Wolf, a mini-game that shows how well different devices handle games, and VR Demo, a virtual reality prototype workload based on a room escape scenario.

This fall, a team of NC State students is developing a software console for automating machine learning tests. Ideally, the tool will let future testers specify custom workload combinations, compute a performance metric, and upload results to our database. The project will also assess the impact of the framework on performance scores. In fact, the console will perform many of the same functions we plan to implement with AIXPRT.

The students have worked very hard on the project, and have learned quite a bit about benchmarking practices and several new software tools. The project will wrap up in the next couple of weeks, and we’ll share additional details as soon as possible. Early next year, we’ll publish a video about the experience.

If you’d like to join the NC State students and hundreds of other XPRT community members in the future of benchmark development, please let us know!

Justin

Posted in AI, AIXPRT, Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Education, Future of performance evaluation, Nebula Wolf, North Carolina State University, Virtual reality, VR Demo | Tagged AI, benchmarks, computer science, gaming, machine learning, NC State, NCSU, North Carolina, North Carolina State University, university, virtual reality |

HDXPRT 4: Getting it right

By Justin Greene

on November 8, 2018

For BenchmarkXPRT Development Community members anticipating the HDXPRT 4 Community Preview (CP), we want to thank you for your patience and explain where we are in the release process.

This past month has brought a flurry of activity in the Windows 10 development world. We’ve been testing HDXPRT 4 extensively on each of the new prerelease builds available through the Windows Insider Program. While testing on a recent Windows 10 Redstone 5 preview build, we began to see inconsistent HDXPRT 4 workload scores on some systems. The difference between those workload scores and scores on the same systems with previous Windows 10 builds was significant enough for us to decide that the best course of action is to hold off on the CP until we understand the issue. We don’t want to release a CP only to run into serious problems with an imminent Windows release. We want to take the time to figure out what’s going on and get it right.

We hope to resolve these issues and publish the HDXPRT 4 CP as soon as possible. Thanks again for your patience. We’ll update the community soon with more information on the anticipated release schedule.

Justin

Posted in Application-based benchmarks, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, HDXPRT, HDXPRT 4, HDXPRT development process, HDXPRT release cycle, Performance benchmarking, Windows 10 | Tagged BenchmarkXPRT Development Community, Community Preview, HDXPRT, HDXPRT 4, Windows, Windows 10 |

A new BatteryXPRT 2014 for Android build is available

By Justin Greene

on October 26, 2018

In last week’s blog, we discussed why we now consider full BatteryXPRT rundown tests to be the most accurate and why we’re releasing a new build (v110) that increases the default BatteryXPRT test from 5.25 hours (seven iterations) to 45 hours (60 iterations). We also built v110 using Android Studio SDK 27, in order to bring BatteryXPRT up to date with current Android standards. Today, we’ve posted the new build on BatteryXPRT.com and in the Google Play Store, and we’ve also published an updated user manual. Please contact us if you have any questions about BatteryXPRT testing.

Justin

Posted in Android, Battery life, BatteryXPRT 2014 for Android, Benchmarking, BenchmarkXPRT | Tagged Android, battery life, BatteryXPRT, mobile, Performance, phones, tablets |

The new WebXPRT 3 Processor Comparison Chart

By Justin Greene

on October 25, 2018

Last fall, we published the WebXPRT 2015 Processor Comparison Chart, a tool that makes it easier to access hundreds of PT-curated, real-world performance scores from a wide range of devices including everything from TVs to phones to tablets to PCs. Today, we’re happy to announce that we’ve added a WebXPRT 3 Processor Comparison Chart.

The WebXPRT 3 chart follows the same format as the WebXPRT 2015 chart, letting you click the average score of each processor to view all the WebXPRT 3 results we currently have for that processor. You can change the number of results the chart displays on each page, and as the screenshot below shows, a new drop-down menu lets you toggle back and forth between the WebXPRT 3 and WebXPRT 2015 charts. W plan to add additional capabilities on a regular basis, so if you have ideas for features and types of data you’d like to see, let us know!

If you’d like to submit results for us to consider for publication in the chart, follow the detailed instructions here. The submission process is quick and easy. We look forward to seeing your results!

Justin

Posted in Android, Battery life, BatteryXPRT 2014 for Android, Browser-based benchmarks, Performance benchmarking, WebXPRT, WebXPRT 3, WebXPRT Processor Comparison Chart | Tagged Android, battery life, BatteryXPRT, Performance, processor, PT, WebXPRT |

A BatteryXPRT bug fix is on the way

By Justin Greene

on October 18, 2018

Some time ago, we started to see unusual BatteryXPRT battery life estimates and high variance on some devices when running tests at the default length of 5.25 hours (seven 45-minute iterations). We suspected that the problem resulted from changes in how new OS versions report battery life on certain devices (e.g., charging past a reported level of 100 percent). In addition, the progress of battery technology in general means that the average phone battery lasts much longer than it did a few years ago. Together, these factors sometimes led to BatteryXPRT runs where the OS reported little to no battery decrease during the first few iterations of a test. We concluded that 5.25 hours wasn’t long enough to produce an accurate battery life estimate.

After extensive experimentation and testing, we’ve decided to release a new build that increases the default BatteryXPRT test length from 5.25 hours (seven iterations) to 45 hours (60 iterations) to allow enough time for a full rundown on most phones. Based on our testing, we consider full rundown tests to be the most accurate and will use those exclusively in our Spotlight testing and elsewhere. Testers will still have the option of choosing shorter test durations, but BatteryXPRT will flag the results with a qualifier that recommends performing a full rundown.

We plan to release the updated build by the end of next week and update BatteryXPRT documentation to reflect the changes. We have not changed any of the workloads and both performance results and full-rundown battery life estimates will be comparable to results from earlier builds.

BatteryXPRT continues to be a useful tool for gauging the performance and expected battery life of Android devices while simulating real-world tasks. If you have any questions about BatteryXPRT, please feel free to ask!

Justin

Posted in Android, Battery life, BatteryXPRT 2014 for Android, Benchmarking, BenchmarkXPRT, Mobile devices, Performance benchmarking | Tagged Android, battery life, BatteryXPRT, mobile devices, XPRT Spotlight |

Category: Benchmarking

The MobileXPRT 3 Community Preview is almost here!

XPRT collaborations: North Carolina State University

HDXPRT 4: Getting it right

A new BatteryXPRT 2014 for Android build is available

The new WebXPRT 3 Processor Comparison Chart

A BatteryXPRT bug fix is on the way

Check out the other XPRTs: