BenchmarkXPRT Blog banner

Category: Performance benchmarking

AIXPRT Community Preview 2 is almost here!

Justin Greene
By Justin Greene
on May 9, 2019

In last week’s blog, we predicted that the second AIXPRT Community Preview (CP2) would be ready for release later this month. Since then, the development process has accelerated, and we now expect to release CP2 as early as tomorrow, May 10.

Those who have access to the existing AIXPRT Community Preview GitHub repository will be able to access CP2 the same way as before. In addition to making the build available on GitHub, we’ll also post CP2 on an AIXPRT tab in the XPRT Members’ Area (login required). If you don’t have a BenchmarkXPRT Development Community membership, please contact us and we’ll help you register.

Testing with AIXPRT CP2 in Ubuntu will be the same as with the first CP, and none of the CP2 changes will affect results. In Windows, testers will be able to use OpenVINO to target a system’s CPU and GPU, and TensorFlow to target CPUs. We’re still investigating ways to support TensorFlow GPU and TensorFlow-TensorRT testing in Windows.

We’re also continuing to work on the improvements to the AIXPRT results viewer that we mentioned last week. We won’t be able to implement all of the changes by tomorrow, but rather than waiting until we’re finished, we’ll be rolling out improvements as they become ready.

We’ll continue to keep everyone up to date with AIXPRT news here in the blog. If you have any questions or comments, please let us know.

Justin

Posted in AI, AIXPRT, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Cross-platform benchmarks, GitHub, Machine learning, OpenVINO, Performance benchmarking, TensorFlow, TensorRT, Ubuntu | Tagged , , , , , , , , |

An update on AIXPRT development

Bill Catchings
By Bill Catchings
on May 2, 2019

It’s been almost two months since the AIXPRT Community Preview went live, and we want to provide folks with a quick update. Community Preview periods for the XPRTs generally last about a month. Because of the complexity of AIXPRT and some of the feedback we’ve received, we plan to release a second AIXPRT Community Preview (CP2) later this month.

One of the biggest additions in CP2 will be the ability to run AIXPRT on Windows. AIXPRT currently requires test systems to run Ubuntu 16.04 LTS. This is fine for testers accustomed to Linux environments, but presents obstacles for those who want to test in a traditional Windows environment. We will not be changing the tests themselves, so this update will not influence existing results from Ubuntu. We plan to make CP2 available for download from the BenchmarkXPRT website for people who don’t wish to deal with GitHub.

Also, after speaking with testers and learning more about the kinds of data points people are looking for in AIXPRT results, we’ve decided to make significant adjustments to the AIXPRT results viewer. To make it easier for visitors to find what they’re looking for, we’ll add filters for key categories such as batch size, toolkit, and latency percentile (e.g., 50th, 90th, 99th), among others. We’ll also allow users to set desired ranges for metrics such as throughput and latency.

Finally, we’re adding a demo mode that displays some images and other information on the screen while a test is running to give users a better idea what is happening. While we haven’t seen results change while running in demo mode, users should not publish demo results or use them for comparison.

We hope to release CP2 in the second half of May and a GA version in mid-June. However, this project has more uncertainties than we usually encounter with the XPRTs, so that timeline could easily change.

We’ll continue to keep everyone up to date with AIXPRT news here in the blog. As always, we appreciate your suggestions. If you have any questions or comments about AIXPRT, please let us know.

Bill

Posted in AI, AIXPRT, Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Cross-platform benchmarks, Future of performance evaluation, Linux, Machine learning, OpenVINO, Performance benchmarking, TensorFlow, TensorRT, Ubuntu | Tagged , , , , |

A new MobileXPRT 3 build is available

Justin Greene
By Justin Greene
on April 25, 2019

Today, we published an updated MobileXPRT 3 build, version 3.114.2.1, on MobileXPRT.com and in the Google Play Store. The new build fixes an issue that was causing crashes on Xiaomi phones. Xiaomi holds significant market share in China, so we wanted to address the issue as soon as possible.

Xiaomi phones use a proprietary Android-based firmware called MIUI, which requires apps to communicate with the system in a specific way. When we originally built MobileXPRT 3, Android allowed an app’s code to send implicit messages calling certain classes of actions. In MIUI, the code must broadcast explicit messages that call the exact action necessary (e.g., waking from sleep). The requirement can improve security by allowing more granular levels of user control, and save power by restricting the number of unseen tasks that apps can run in the background without a user’s knowledge. The new MobileXPRT 3 build code complies with MIUI’s requirements. Other Android-based platforms will likely require explicit messages in the near future, so we’re hoping the new build will be relatively future proof.

We also fixed a few small UI bugs and improved the accuracy of the system hardware information that the app reports when a user submits a set of results. None of these changes affect performance, so scores from prior MobileXPRT 3 builds are comparable to those from the new build. If you have any questions or comments about MobileXPRT 3, please let us know.

Justin

Posted in Android, Android O, Benchmarking, MobileXPRT, MobileXPRT 3, MobileXPRT 3, Performance benchmarking, Phones, Tablets | Tagged , , , , , , |

A new playing field for WebXPRT

Justin Greene
By Justin Greene
on April 18, 2019

WebXPRT is one of the go-to benchmarks for evaluating browser performance, so we’re always interested in browser development news. Recently, Microsoft created a development channel where anyone can download early versions of an all-new Microsoft Edge browser. Unlike previous versions of Edge, Microsoft constructed the new browser using the Chromium open-source project, the same foundation underlying the Google Chrome browser and Chrome OS.

One interesting aspect of the new Edge development strategy is the changes that Microsoft is making to more than 50 services that Chromium has included. If you use Chrome daily, you’ve likely become accustomed to certain built-in services such as ad block, spellcheck, translate, maps integration, and form fill, among many others. While each of these is useful, a large number of background services running simultaneously can slow browsing and sap battery life. In the new Edge, Microsoft is either reworking each service or removing it altogether, with the hope of winning users by providing a cleaner, faster, and more power-efficient experience. You can read more about Microsoft’s goals for the new project on the Microsoft Edge Insider site.

As we’ve discussed before, many factors contribute to the speed of a browsing experience and its WebXPRT score. It’s too early to know how the new Microsoft Edge will stack up against other browsers, but when the full version comes out of development, you can be sure that we’ll be publishing some comparison scores. I’ve installed the Dev Channel version of Edge on my personal machine and run WebXPRT 3. While I can’t publish the scores from this early version, I can tell you that the results were interesting. Have you run WebXPRT 3 on the new Microsoft Edge? How do you think it compares to competitors? We’d love to hear your thoughts.

Justin

Posted in Benchmarking, Browser-based benchmarks, Chrome OS, Cross-platform benchmarks, Google, Google Chrome, Microsoft Edge, Performance benchmarking, WebXPRT, WebXPRT 3 | Tagged , , , , , , , , |

BatteryXPRT provides the objective battery life data that shoppers need

Justin Greene
By Justin Greene
on April 11, 2019

Over the last few weeks, we’ve discussed the capabilities and benefits of TouchXPRT and CrXPRT. This week, we’d like to reintroduce readers to BatteryXPRT, our app that evaluates the battery life and performance of Android devices.

Battery life for phones and tablets has improved dramatically over the last several years, to the point where many devices can support continuous use for well over a full work day on a single charge. This improvement is the result of advances in battery hardware technology, increased processor efficiency, and smarter utilization of software services by the operating system. Battery life has increased to some extent for most device categories and price points. However, enough of a range remains between devices at each level that access to objective battery life data is valuable for device shoppers.

Without BatteryXPRT, shoppers must rely on manufacturer estimates or full rundown tests that don’t resemble the types of things we do with our phones and tablets every day. A rundown test that surfs the web continuously for over 15 hours reveals which devices last the longest performing that specific task. It doesn’t tell you which devices last the longest over a full day performing a variety of common activities such as web browsing, watching videos, browsing and editing photos, playing music, and periodically sleeping. During BatteryXPRT’s battery life test, the app executes those same types of tasks and produces a performance score based on the speed with which a device completes each task.

BatteryXPRT provides an intuitive user interface in English and Simplified Chinese, and easy-to-understand results for both battery life and performance. Because your data connection can have a significant effect on battery life, BatteryXPRT runs in airplane mode, connected to the Internet via Wi-Fi, or connected to the Internet through a cellular data connection.

BatteryXPRT is easy to install and run, and is a great resource for anyone who wants to evaluate how well an Android device will meet their needs. If you’d like to see test results from a variety of Android devices, go to BatteryXPRT.com and click View Results, where you’ll find scores from many different Android devices.

If you’d like to run BatteryXPRT

Simply download BatteryXPRT from the Google Play store or BatteryXPRT.com. The BatteryXPRT installation instructions and user manual provide step-by-step instructions for configuring your device and kicking off a test. We designed BatteryXPRT to be compatible with a wide variety of Android devices, but because there are so many devices on the market, it is inevitable that users occasionally run into problems. In the Tips, tricks, and known issues document, we provide troubleshooting suggestions for issues we encountered during development testing.

If you’d like to learn more

The Exploring BatteryXPRT 2014 for Android white paper covers almost every aspect of the benchmark. In it, we explain the guiding concepts behind BatteryXPRT’s development, as well as the benchmark’s structure. We describe the component tests, the differences between the app’s Airplane and Network/Wi-Fi modes, and the statistical processes used to calculate expected battery life.

Justin

Posted in Android, Battery life, BatteryXPRT 2014 for Android, Benchmark metrics, Benchmarking, Google, Performance benchmarking, Phones, What makes a good benchmark? | Tagged , , , , , |

TouchXPRT: a great tool for evaluating Windows performance

Justin Greene
By Justin Greene
on April 4, 2019

From time to time, we remember that some XPRT users have experience with only one or two of the benchmark tools in our portfolio. They might have bookmarked a link to WebXPRT they found in a tech review or copied the HDXPRT installer package from a flash drive in their lab, but are unaware of other members of the XPRT family that could be useful to them. To spread the word on the range of capabilities the XPRTs offer, we occasionally highlight one of the XPRT tools in the blog . Last week, we discussed CrXPRT, a benchmark for evaluating the performance and battery life of Chrome OS devices. Today, we focus on TouchXPRT, our app for evaluating the performance of Windows 10 devices.

While our first benchmark, HDXPRT, is a great tool for assessing how well Windows machines handle media creation tasks using real commercial applications, it’s simply too large to run on most Windows tablets, 2-in-1s, and laptops with limited memory. To test those devices, we developed the latest version of TouchXPRT as a Universal Windows Platform app. As a Windows app, installing TouchXPRT is easy and quick (about 15 minutes). It runs five tests that simulate common photo, video, and music editing tasks; measures how quickly the device completes each of those tasks; and provides an overall score. It takes about 15 minutes to run on most devices. Labs can also automate testing using the command line or a script.

Want to run TouchXPRT?

Download TouchXPRT from the Microsoft Store or from TouchXPRT.com. The TouchXPRT 2016 release notes provide step-by-step instructions. To compare device scores, go to the TouchXPRT 2016 results page, where you’ll find scores from many Windows 10 devices.

Want to dig into the details?

Check out the Exploring TouchXPRT 2016 white paper. In it, we discuss the TouchXPRT development process, its component tests and workloads, and how it calculates individual workload and overall scores. We also provide instructions for automated testing.

BenchmarkXPRT Development Community members also have access to the TouchXPRT source code, so consider joining the community today. There’s no obligation and membership is free for members of any company or organization with an interest in benchmarks.

If you’ve been looking for a Windows performance evaluation tool that’s easy to use and has the flexibility of a UWP app, give TouchXPRT a try and let us know what you think!

Justin

Posted in Battery life, Benchmark metrics, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Chrome OS, CrXPRT, HDXPRT, Performance benchmarking, TouchXPRT, TouchXPRT 2016, TouchXPRT release cycle, TouchXPRT results, WebXPRT, Windows 10 | Tagged , , , , , , , , , |

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2024   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x