Category: Performance benchmarking

A new HDXPRT 2014 build is available

on February 9, 2017

Last fall, we identified a way to run HDXPRT 2014, originally developed for Windows 8, on Windows 10. The method involved overwriting the HDXPRT CPU-Z files with newer versions and performing a few additional pre-test configuration steps. You can read more details about those steps here.

Today, we’re releasing a new build of HDXPRT 2014 (v1.2) that eliminates the need to overwrite the CPU-Z files. The new build is available for download at HDXPRT.com. Please note that the app package is 5.08 GB, so allow time and space for the download process.

We also updated the HDXPRT 2014 User Manual to reflect changes in pre-test system configuration and to include the settings we recommend for newer builds of Windows 10.

The changes in the new build do not affect results, so v1.2 scores are comparable to v1.1 scores on the same system.

The new build ran well during testing in our labs, but issues could emerge as Microsoft releases new Windows updates. If you have any questions about HDXPRT or encounter any issues during testing, we encourage you to let us know.

We look forward to seeing your test results!

Justin

Posted in Application-based benchmarks, Collaborative benchmark development, HDXPRT, HDXPRT 2014, Performance benchmarking, Windows 10 |

BatteryXPRT 2014 gets an update

By Justin Greene

on January 26, 2017

After Android 7 (Nougat) was released on select devices this past fall, we discovered an issue with BatteryXPRT on devices running Android 7 and above. The battery life tests were completing accurately and reliably, but the test was not producing a performance score.

The problem was a result of significant changes in the Android development environment. Android 7 restricted the flags used for different target architectures when linking native code components, and that caused issues while executing part of the Create Slideshow workload. We resolved the issue by changing the linked flags. Also, we migrated the BatteryXPRT code from the Eclipse and Android SDK development environments to the up-to-date Android Studio environment. This allowed us to rebuild the app in a way that maintains compatibility with the most recent versions of Android.

Today, we’re releasing a new build of BatteryXPRT 2014 (v104) at BatteryXPRT.com and the Google Play store. Scores from this build are comparable with previous BatteryXPRT scores, and if you’re testing with a version of BatteryXPRT that you downloaded from the Google Play store, you should receive the new build via an app update.

Click here to download the new BatteryXPRT installer (330 MB) directly from our site.

For users who have limited bandwidth or trouble accessing the Google Play store, downloading the APK files (26.7 MB total) may make installation easier.

Download the updated BatteryXPRT APK (2.8 MB) directly from our site.

Download the updated BatteryXPRT Tests APK (23.9 MB) directly from our site.

If you have any questions about the update or any other XPRT-related topic, feel free to contact us at BenchmarkXPRTSupport@principledtechnologies.com.

Justin

Posted in Android, Android N, Battery life, BatteryXPRT 2014 for Android, Performance benchmarking |

CrXPRT’s future

By Justin Greene

on January 19, 2017

This week, we’re continuing our review of the XPRT portfolio by discussing the future of CrXPRT. CrXPRT, designed for use with Chrome OS, is a tool for evaluating the performance and battery life of Chromebooks as they handle everyday tasks. The app’s performance test, which measures Chromebook speed, produces an overall score and individual scores for each workload. The battery life test produces an estimated battery life and a separate performance score. CrXPRT is easy to install and use, and like BatteryXPRT, it evaluates battery life in half a workday.

We developed CrXPRT in response to the growing popularity of Chromebooks, especially in the education sector. The number of OEMs manufacturing Chromebooks has grown dramatically, along with the range of Chromebook price points and form factors. That growth shows no signs of slowing down, so CrXPRT is more relevant than ever as a tool for helping consumers make informed buying decisions.

As Chromebook market share continues to grow, however, it’s clear that significant changes to the Chrome OS environment are on the way. One big change is Google’s decision to bring Android apps, and the Google Play store itself, to Chrome OS. Another change is the plan to “begin the evolution away” from the Chrome apps platform and phase out Chrome app support on other platforms within the next two years.

There are also reports of a hybrid Android-Chrome OS operating system. Codenamed “Andromeda,” it would unite the Android and Chrome OS environments in a manner similar to the way Microsoft Continuum allows Windows 10 to run on a wide variety of device types. Details on Andromeda are few and far between, but it would obviously be a game changer.

The Google Play store rollout to select Chromebooks is already well underway. As for the other changes, it remains to be seen exactly when and how they will be implemented. The Chromium team did state that all types of Chrome apps will remain supported and maintained on Chrome OS for the foreseeable future.

For us, it’s important to maintain the ability to measure both performance and battery life on Chromebooks. The current version of CrXPRT does the job well, so we don’t see a need for a new version until the situation becomes more clear. In the meantime, we’ll be keeping an eye on Chrome-related news.

As always, we’re interested in your feedback. If you have any thoughts on CrXPRT 2015 or the future of Chromebook evaluation, let us know!

Justin

Posted in Android, BatteryXPRT 2014 for Android, Chrome OS, Chromebooks, CrXPRT, Performance benchmarking, Windows 10 |

WebXPRT in 2017

By Bill Catchings

on December 8, 2016

Over the last few weeks, we’ve discussed the future of HDXPRT and BatteryXPRT. This week, we’re discussing what’s in store for WebXPRT in 2017.

WebXPRT is our most popular tool. Manufacturers, developers, consumers, and media outlets in more than 350 cities and 57 countries have run WebXPRT over 113,000 times to date. The benchmark runs quickly and simply in most browsers and produces easy-to-understand results that are useful for comparing web browsing performance across a wide variety of devices and browsers. People love the fact that WebXPRT runs on almost any platform that has a web browser, from PCs to phones to game consoles.

More people are using WebXPRT in more places and in more ways than ever before. It’s an unquestioned success, but we think this is a good time to make it even better by beginning work on WebXPRT 2017. Any time change comes to a popular product, there’s a risk that faithful fans will lose the features and functionality they’ve grown to love. Relevant workloads, ease of use, and extensive compatibility have always been the core components of WebXPRT’s success, so we want to reassure users that we’re committed to maintaining all of those in future versions.

Some steps in the WebXPRT 2017 process are straightforward, such as the need to reassess the existing workload lineup and update content to reflect advances in commonly used technologies. Other steps, such as introducing new workloads to test emerging browser technologies, may be tricky to implement, but could offer tremendous value in the months and years ahead.

Are there test scenarios or browser technologies you would like to see in WebXPRT 2017, or tests you think we should get rid of? Please let us know. We want to hear from you and make sure that we’re crafting a performance tool that continues to meet your needs.

Bill

Posted in BatteryXPRT 2014 for Android, Future of performance evaluation, HDXPRT, HTML5, Performance benchmarking, Web-based testing, WebXPRT, WebXPRT 2017 |

HDXPRT’s future

By Bill Catchings

on November 3, 2016

While industry pundits have written many words about the death of the PC, Windows PCs are going through a renaissance. No longer do you just choose between a desktop or a laptop in beige or black. There has been an explosion of choices.

Whether you want a super-thin notebook, a tablet, or a two-in-one device, the market has something to offer. Desktop systems can be small devices on your desk, all-in-ones with the PC built into the monitor, or old-style boxes that sit on the floor. You can go with something inexpensive that will be sufficient for many tasks or invest in a super-powerful PC capable of driving today’s latest VR devices. Or you can get a new Microsoft Surface Studio, an example of the new types of devices entering the PC scene.

The current proliferation of PC choices means that tools that help buyers understand the performance differences between systems are more important than they have been in years. Because HDXPRT is one such tool, we expect demand for it to increase.

We have many tasks ahead of us as we prepare for this increased demand. The first is to release a version of HDXPRT 2014 that doesn’t require a patch. We are working on that and should have something ready later this month.

For the other tasks, we need your input. We believe we need to update HDXPRT to reflect the world of high-definition content. It’s tempting to simply change the name to UHDXPRT, but this was our first XPRT and I’m partial to the original name. How about you?

As far as tests, what should a 2017 version of HDXPRT include? We think 4K-related workloads are a must, but aren’t sure whether 4K playback tests are the way to go. What do you think? We need to update other content, such as photo and video resolutions, and replace outdated applications with current versions. Would a VR test would be worthwhile?

Please share your thoughts with us over the coming weeks as we put together a plan for the next version of HDXPRT!

Bill

Posted in Application-based benchmarks, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, HDXPRT, HDXPRT development process, Performance benchmarking, What makes a good benchmark?, Windows 10 |

Tracking device evolution with WebXPRT ’15, part 2

By Justin Greene

on October 27, 2016

Last week, we used the Apple iPhone as a test case to show how hardware advances are often reflected in benchmark scores over time. When we compared WebXPRT 2015 scores for various iPhone models, we saw a clear trend of progressively higher scores as we moved from phones with an A7 chip to phones with A8, A9, and A10 Fusion chips. Performance increases over time are not surprising, but WebXPRT ’15 scores also showed us that upgrading from an iPhone 6 to an iPhone 6s is likely to have a much greater impact on web-browsing performance than upgrading from an iPhone 6s to an iPhone 7.

This week, we’re revisiting our iPhone test case to see how software updates can boost device performance without any changes in hardware. The original WebXPRT ’15 tests for the iPhone 5s ran on iOS 8.3, and the original tests for the iPhone 6s, 6s Plus, and SE ran on variants of iOS 9. We updated each phone to iOS 10.0.2 and ran several iterations of WebXPRT ’15.

Upgrading from iOS 8.3 to iOS 10 on the iPhone 5s caused a 17% increase in web-browsing performance, as measured by WebXPRT. Upgrading from iOS 9 to iOS 10 on the iPhone 6s, 6s Plus, and SE produced web-browsing performance gains of 2.6%, 3.6%, and 3.1%, respectively.

The chart below shows the WebXPRT ’15 scores for a range of iPhones, with each iPhone’s iOS version upgrade noted in parentheses. The dark blue columns on the left represent the original scores, and the light blue columns on the right represent the upgrade scores.

As with our hardware comparison last week, these scores are the median of a range of scores for each device in our database. These scores come both from our own testing and from device reviews from popular tech media outlets.

These results reinforce a message that we repeat often, that many factors other than hardware influence performance. Designing benchmarks that deliver relevant and reliable scores requires taking all factors into account.

What insights have you gained recently from WebXPRT ’15 testing? Let us know!

Justin

Posted in Apple, Application-based benchmarks, Benchmark metrics, Benchmarking, Browser-based benchmarks, iOS, Performance benchmarking, Uncategorized, WebXPRT |

Category: Performance benchmarking

A new HDXPRT 2014 build is available

BatteryXPRT 2014 gets an update

CrXPRT’s future

WebXPRT in 2017

HDXPRT’s future

Tracking device evolution with WebXPRT ’15, part 2

Check out the other XPRTs: