Category: Benchmarking

Learning something new every day

on June 1, 2017

We’re constantly learning and thinking about how the XPRTs can help people evaluate the tech that will soon be a part of daily life. It’s why we started work on a tool to evaluate machine learning capabilities, and it’s why we developed CrXPRT in response to Chromebooks’ growing share of the education sector.

The learning process often involves a lot of tinkering in the lab, and we recently began experimenting with Neverware’s CloudReady OS. CloudReady is an operating system based on the open-source Chromium OS. Unlike Chrome OS, which can run on only Chromebooks, CloudReady can run on many types of systems, including older Windows and OS X machines. The idea is that individuals and organizations can breathe new life into aging hardware by incorporating it into a larger pool of devices managed through a Google Admin Console.

We were curious to see if it worked as advertised, and if it would run CrXPRT 2015. Installing CloudReady on an old Dell Latitude E6430 was easy enough, and we then installed CrXPRT from the Chrome Web Store. Performance tests ran without a hitch. Battery life tests would kick off but not complete, which was not a big surprise because the battery life calls involved were developed specifically for Chrome OS.

So, what role can CrXPRT play with CloudReady, and what are the limitations? CloudReady has a lot in common with Chrome OS, but there are some key differences. One way we see the CrXPRT performance test being useful is for comparing CloudReady devices. Say that an organization was considering adopting CloudReady on certain legacy systems but not on others; CrXPRT performance scores would provide insight into which devices performed better with CloudReady. While you could use CrXPRT to compare those devices to Chromebooks, the differences between the operating systems are significant enough that we cannot guarantee the comparison would be a fair one.

Have you spent any time working with CloudReady, or are there other interesting new technologies you’d like us to investigate? Let us know!

Justin

Posted in Battery life, Benchmark metrics, Chrome OS, Chromebooks, CrXPRT, Performance benchmarking |

BatteryXPRT: A quick and reliable way to estimate Android battery life

By Justin Greene

on May 25, 2017

In the last few weeks, we reintroduced readers to the capabilities and benefits of TouchXPRT and CrXPRT. This week, we’d like to reintroduce BatteryXPRT 2014 for Android, an app that evaluates the battery life and performance of Android devices.

When purchasing a phone or tablet, it’s good to know how long the battery will last on a typical day and how often you’ll need to charge it. Before BatteryXPRT, you had to rely on a manufacturer’s estimate or full rundown tests that perform tasks that don’t resemble the types of things we do with our phones and tablets every day.

We developed BatteryXPRT to estimate battery life reliably in just over five hours, so testers can complete a full evaluation in one work day or while sleeping. You can configure it to run while the device is connected to a network or in Airplane mode. The test also produces a performance score by running workloads that represent common everyday tasks.

BatteryXPRT is easy to install and run, and is a great resource for anyone who wants to evaluate how well an Android device will meet their needs. If you’d like to see test results from a variety of Android devices, go to BatteryXPRT.com and click View Results, where you’ll find scores from many different Android devices.

If you’d like to run BatteryXPRT:

Simply download BatteryXPRT from the Google Play store or BatteryXPRT.com. The BatteryXPRT installation instructions and user manual provide step-by-step instructions for how to configure your device and kick off a test. We designed BatteryXPRT 2014 for Android to be compatible with a wide variety of Android devices, but because there are so many devices on the market, it is inevitable that users occasionally run into problems. In the Tips, tricks, and known issues document, we provide troubleshooting suggestions for issues we encountered during development testing.

If you’d like to learn more:

We offer a full online BatteryXPRT training course that covers almost every aspect of the benchmark. You can view the sections in order or jump to the parts that interest you. We guarantee that you’ll learn something new!

If you’d like to dig into the details:

Check out the Exploring BatteryXPRT 2014 for Android white paper. In it, we discuss the app’s development and structure. We also describe the component tests; explain the differences between the test’s Airplane, Wi-Fi, and Cellular modes; and detail the statistical processes we use to calculate expected battery life.

If you’d like to dig even deeper, the BatteryXPRT source code is available to members of the BenchmarkXPRT Development Community, so consider joining today. Membership is free for members of any company or organization with an interest in benchmarks, and there are no obligations after joining.

If you haven’t used BatteryXPRT before, try it out and let us know what you think!

Justin

Posted in Android, Battery life, BatteryXPRT 2014 for Android, Benchmark metrics, Benchmarking, BenchmarkXPRT development community, CrXPRT, TouchXPRT, What makes a good benchmark? |

CrXPRT: a valuable tool for evaluating Chromebooks

By Justin Greene

on May 11, 2017

Last week, we reintroduced readers to TouchXPRT 2016. This week, we invite you to get to know CrXPRT, an app for Chrome OS devices.

When you buy a Chromebook, it’s important to know how long the battery will last on a typical day and how well it can handle everyday tasks. We developed CrXPRT 2015 to help answer those questions. CrXPRT measures how fast a Chromebook handles everyday tasks such as playing video games, watching movies, editing pictures, and doing homework, and it also measures battery life. The performance test, which measures the speed of your Chromebook, gives you individual workload scores and an overall score. The battery life test produces an estimated battery life time, a separate performance score, and a frames-per-second (FPS) rate for a built-in HTML5 gaming component.

CrXPRT completes the battery life evaluation in half a workday, and delivers results you can understand. Before CrXPRT, you had to rely on the manufacturer’s performance claims and estimated battery life. Now, CrXPRT provides an objective evaluation tool that’s easy to use for anyone interested in Chromebooks. To learn more about CrXPRT, check out the links below.

Watch CrXPRT in action:

To test your Chromebook’s performance or battery life:

Simply download CrXPRT from the Chrome Web Store. Installation is quick and easy, and the CrXPRT 2015 user manual provides step-by-step instructions. A typical performance test takes about 15 minutes, and a battery life test will take 3.5 hours once the system is charged and configured for testing. If you’d like to see how your score compares to other Chromebooks, visit the CrXPRT results page.

If you’d like to dig into the details:

Read the Exploring CrXPRT 2015 white paper. In it, we discuss the concepts behind CrXPRT 2015, its development process, and the application’s structure. We also describe the component tests and explain the statistical processes used to calculate expected battery life.

BenchmarkXPRT Development Community members also have access to the CrXPRT source code, so if you’re interested, join today! There’s no obligation and membership is free for members of any company or organization with an interest in benchmarks.

If you have a Chromebook you’d like to evaluate, give CrXPRT a try and let us know what you think!

Justin

Posted in Battery life, BenchmarkXPRT development community, Chrome OS, Chromebooks, CrXPRT, Education, Google Chrome, Performance benchmarking |

Getting to know TouchXPRT

By Justin Greene

on May 4, 2017

Many of our community members first encountered the XPRTs when reading about WebXPRT or MobileXPRT in a device review, using TouchXPRT or HDXPRT in an OEM lab, or using BatteryXPRT or CrXPRT to evaluate devices for bulk purchasing on behalf of a corporation or government agency. They know that specific XPRT provided great value in that context, but may not know about the other members of the XPRT family.

To help keep folks up to date on the full extent of XPRT capabilities, we like to occasionally “reintroduce” each of the XPRTs. This week, we invite you to get to know TouchXPRT.

We developed TouchXPRT 2016 as a Universal Windows Platform app for Windows 10. We wanted to offer a free tool that would provide consumers with objective information about how well a Windows 10 or Windows 10 Mobile laptop, tablet, or phone handles common media tasks. To do this, TouchXPRT runs five tests that simulate the kinds of photo, video, and music editing tasks people do every day. It measures how quickly the device completes each of those tasks and provides an overall score. To compare device scores, go to TouchXPRT.com and click View Results, where you’ll find scores from many different Windows 10 and Windows 10 Mobile devices.

TouchXPRT is easy to install and run, and is a great resource for anyone who wants to evaluate the performance of a Windows 10 device.

If you’d like to run TouchXPRT:

Simply download TouchXPRT from the Microsoft Store. (If that doesn’t work for you, you can also download it directly from TouchXPRT.com.) Installing it should take about 15 minutes, and the TouchXPRT 2016 release notes provide step-by-step instructions.

If you’d like to dig into the details:

Check out the Exploring TouchXPRT 2016 white paper. In it, we discuss the TouchXPRT development process, its component tests and workloads, and how it calculates individual workload and overall scores. We also provide instructions for automated testing.

BenchmarkXPRT Development Community members also have access to the TouchXPRT source code, so consider joining today. There’s no obligation and membership is free for members of any company or organization with an interest in benchmarks.

If you haven’t tried running TouchXPRT before, give it a shot and let us know what you think!

Justin

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Mobile devices, Performance benchmarking, Performance testing on tablets, TouchXPRT, TouchXPRT 2016, TouchXPRT development process, TouchXPRT results |

Learning about machine learning

By Bill Catchings

on April 27, 2017

Everywhere we look, machine learning is in the news. It’s driving cars and beating the world’s best Go players. Whether we are aware of it or not, it’s in our lives–understanding our voices and identifying our pictures.

Our goal of being able to measure the performance of hardware and software that does machine learning seems more relevant than ever. Our challenge is to scan the vast landscape that is machine learning, and identify which elements to measure first.

There is a natural temptation to see machine learning as being all about neural networks such as AlexNet and GoogLeNet. However, new innovations appear all the time and lots of important work with more classic machine learning techniques is also underway. (Classic machine learning being anything more than a few years old!) Recursive neural networks used for language translation, reinforcement learning used in robotics, and support vector machine (SVM) learning used in text recognition are just a few examples among the wide array of algorithms to consider.

Creating a benchmark or set of benchmarks to cover all those areas, however, is unlikely to be possible. Certainly, creating such an ambitious tool would take so long that it would be of limited usefulness.

Our current thinking is to begin with a small set of representative algorithms. The challenge, of course, is identifying them. That’s where you come in. What would you like to start with?

We anticipate that the benchmark will focus on the types of inference learning and light training that are likely to occur on edge devices. Extensive training with large datasets takes place in data centers or on systems with extraordinary computing capabilities. We’re interested in use cases that will stress the local processing power of everyday devices.

We are, of course, reaching out to folks in the machine learning field—including those in academia, those who create the underlying hardware and software, and those who make the products that rely on that hardware and software.

What do you think?

Bill

Posted in AI, AR, Augmented reality, Automated cars, Automation, computer vision, Future of performance evaluation, Machine learning, What makes a good benchmark? |

Evolve or die

By Eric Hale

on April 20, 2017

Last week, Google announced that it would retire its Octane benchmark. Their announcement explains that they designed Octane to spur improvement in JavaScript performance, and while it did just that when it was first released, those improvements have plateaued in recent years. They also note that there are some operations in Octane that optimize Octane scores but do not reflect real-world scenarios. That’s unfortunate, because they, like most of us, want improvements in benchmark scores to mean improvements in end-user experience.

WebXPRT comes at the web performance issue differently. While Octane’s goal was to improve JavaScript performance, the purpose of WebXPRT is to measure performance from the end user’s perspective. By doing the types of work real people do, WebXPRT doesn’t measure only improvements in JavaScript performance; it also measures the quality of the real-world user experience. WebXPRT’s results also reflect the performance of the entire device and software stack, not just the performance of the JavaScript interpreter.

Google’s announcement reminds us that benchmarks have finite life spans, that they must constantly evolve to keep pace with changes in technology, or they will become useless. To make sure the XPRT benchmarks do just that, we are always looking at how people use their devices and developing workloads that reflect their actions. This is a core element of the XPRT philosophy.

As we mentioned last week, we’ve working on the next version of WebXPRT. If you have any thoughts about how it should evolve, let us know!

Eric

Posted in Benchmark metrics, Benchmarking, BenchmarkXPRT development community, Browser-based benchmarks, Future of performance evaluation, Google, JavaScript, Performance benchmarking, What makes a good benchmark? |

Category: Benchmarking

Learning something new every day

BatteryXPRT: A quick and reliable way to estimate Android battery life

If you’d like to run BatteryXPRT:

If you’d like to learn more:

If you’d like to dig into the details:

CrXPRT: a valuable tool for evaluating Chromebooks

Watch CrXPRT in action:

To test your Chromebook’s performance or battery life:

If you’d like to dig into the details:

Getting to know TouchXPRT

If you’d like to run TouchXPRT:

If you’d like to dig into the details:

Learning about machine learning

Evolve or die

Check out the other XPRTs: