Category: Collaborative benchmark development

Planning for the next CrXPRT

on December 5, 2019

We’re currently planning the next version of CrXPRT, our benchmark that evaluates the performance and battery life of Chromebooks. If you’re unfamiliar with CrXPRT, you can find out more about how it works both here in the blog and at CrXPRT.com. If you’ve used CrXPRT, we’d love to hear any suggestions you may have. What do you like or dislike about CrXPRT? What features do you hope to see in a new version?

When we begin work on a new version of any benchmark, one of our first steps is to determine whether the workloads will provide value during the years ahead. As technology and user behavior evolve, we update test content to be more relevant. One example is when we replace photos with ones that use more contemporary file resolutions and sizes.

Sometimes the changing tech landscape prompts us to remove entire workloads and add new ones. The Photo Collage workload in CrXPRT uses Portable Native Client (PNaCl) technology, for which the Chrome team will soon end support. CrXPRT 2015 has a workaround for this issue, but the best course of action for the next version of CrXPRT will be to remove this workload altogether.

The battery life test will also change. Earlier this year, we started to see unusual battery life estimates and high variance when running tests at CrXPRT’s default battery life test length of 3.5 hours, so we’ve been recommending that users perform full rundowns instead. In the next CrXPRT, the battery life test will require full rundowns.

We’ll also be revamping the CrXPRT UI to improve the look of the benchmark and make it easier to use, as we’ve done with the other recent XPRT releases.

We really do want to hear your ideas, and any feedback you send has a chance to shape the future of the benchmark. Let us know what you think!

Justin

Posted in Battery life, benchmark, Benchmarking, BenchmarkXPRT, Browser-based benchmarks, Chrome OS, Chromebooks, Collaborative benchmark development, CrXPRT, Google Chrome, Performance benchmarking | Tagged battery life, benchmark, Chrome, Chrome OS, Chromebooks, CrXPRT |

How to use alternate configuration files with AIXPRT

By Justin Greene

on October 10, 2019

In last week’s AIXPRT Community Preview 3 announcement, we mentioned the new public GitHub repository that we’re using to publish AIXPRT-related information and resources. In addition to the installation readmes for each AIXPRT installation package, the repository contains a selection of alternative test config files that testers can use to quickly and easily change a test’s parameters.

As we discussed in previous blog entries about batch size, levels of precision, and number of concurrent instances, AIXPRT testers can adjust each of these key variables by editing the JSON file in the AIXPRT/Config directory. While the process is straightforward, editing each of the variables in a config file can take some time, and testers don’t always know the appropriate values for their system. To address both of these issues, we are offering a selection of alternative config files that testers can download and drop into the AIXPRT/Config directory.

In the GitHub repository, we’ve organized the available config files first by operating system (Linux_Ubuntu and Windows) and then by vendor (All, Intel, and NVIDIA). Within each section, testers will find preconfigured JSON files set up for several scenarios, such as running with multiple concurrent instances on a system’s CPU or GPU, running with FP32 precision instead of FP16, etc. The picture below shows the preconfigured files that are currently available for systems running Ubuntu on Intel hardware.

Because potential AIXPRT use cases cut across a wide range of hardware segments, including desktops, edge devices, and servers, not all AIXPRT workloads and configs will be applicable to each segment. As we move towards the AIXPRT GA, we’re working to find the best way to parse out these distinctions and communicate them to end users. In many cases, the ideal combination of test configuration variables remains an open question for ongoing research. However, we hope the alternative configuration files will help by giving testers a starting place.

If you experiment with an alternative test configuration file, please note that it should replace the existing default config file. If more than one config file is present, AIXPRT will run all the configurations and generate a separate result for each. More information about the config files and detailed instructions for how to handle the files are available in the EditConfig.md document in the public repository.

We’ll continue to keep everyone up to date with AIXPRT news here in the blog. If you have any questions or comments, please let us know.

Justin

Posted in AI, AIXPRT, benchmark, Benchmark metrics, Collaborative benchmark development, Community Preview, GitHub, Intel, Linux, Machine learning, NVIDIA, Performance benchmarking, Ubuntu | Tagged AI, AIXPRT, batch size, concurrent instances, FP16, FP32, GitHub, Intel, NVIDIA, precision, Windows |

AIXPRT Community Preview 3 is here!

By Justin Greene

on October 3, 2019

We’re happy to announce that the AIXPRT Community Preview 3 (CP3) is now available! As we discussed in last week’s blog, testers can expect three significant changes in AIXPRT CP3:

We updated support for the Ubuntu test packages from Ubuntu version 16.04 LTS to version 18.04 LTS.
We added TensorRT test packages for Windows and Ubuntu. Previously, AIXPRT testers could test only the TensorFlow variant of TensorRT. Now, they can use TensorRT to test systems with NVIDIA GPUs.
We added the Wide and Deep recommender system workload with the MXNet toolkit for Ubuntu systems.

To access AIXPRT CP3, click this access link and submit the brief information form unless you’ve already done so for CP2. You will then gain access to the AIXPRT community preview page. (If you’re not already a BenchmarkXPRT Development Community member, we’ll contact you with more information about your membership.)

On the community preview page, a download table displays the currently available AIXPRT CP3 test packages. Locate the operating system and toolkit you wish to test, and click the corresponding Download link. For detailed installation instructions and information on hardware and software requirements for each package, click the corresponding Readme link. Instead of providing installation guide PDFs as we did for CP2, we are now directing testers to a public GitHub repository. The repository contains the installation readmes for all the test packages, as well as a selection of alternative test configuration files. We’ll discuss the alternative configuration files in more detail in a future blog post.

Note: Those who have access to the existing AIXPRT GitHub repository will be able to access CP3 in the same way as previous versions.

We’ll continue to keep everyone up to date with AIXPRT news here in the blog. If you have any questions or comments, please let us know.

Justin

Posted in AI, AIXPRT, benchmark, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Future of performance evaluation, image classification, Machine learning, MXNet, NVIDIA, object detection, OpenVINO, Performance benchmarking, recommender system, ResNet-50, SSD-MobileNet v1, TensorFlow, TensorRT, Ubuntu, Wide and Deep | Tagged AI, AIXPRT, benchmark, image processing, machine learning, MXNet, object detection, OpenVINO, preview, recommender, TensorFlow, TensorRT, Ubuntu, Windows |

An update on AIXPRT development

By Justin Greene

on September 26, 2019

It’s been a while since we last discussed the AIXPRT Community Preview 3 (CP3) release schedule, so we want to let everyone know where things stand. Testing for CP3 has taken longer than we predicted, but we believe we’re nearly ready for the release.

Testers can expect three significant changes in AIXPRT CP3. First, we updated support for the Ubuntu test packages. During the initial development phase of AIXPRT, Ubuntu version 16.04 LTS (Long Term Support) was the most current LTS version, but version 18.04 is now available.

Second, we have added TensorRT test packages for Windows and Ubuntu. Previously, AIXPRT testers could test only the TensorFlow variant of TensorRT. Now, they can use TensorRT to test systems with NVIDIA GPUs.

Third, we have added the Wide and Deep recommender system workload with the MXNet toolkit. Recommender systems are AI-based information-filtering tools that learn from end user input and behavior patterns and try to present them with optimized outputs that suit their needs and preferences. If you’ve used Netflix, YouTube, or Amazon accounts, you’ve encountered recommender systems that learn from your behavior.

Currently, the recommender system workload in AIXPRT CP3 is available for Ubuntu testing, but not for Windows. Recommender system inference workloads typically run on datacenter hardware, which tends to be Linux based. If enough community members are interested in running the MXNet/Wide and Deep test package on Windows, we can investigate what that would entail. If you’d like to see that option, please let us know.

As always, if you have any questions about the AIXPRT development process, feel free to ask!

Justin

Posted in AI, AIXPRT, benchmark, Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Linux, Machine learning, MXNet, Performance benchmarking, Ubuntu, What makes a good benchmark?, Wide and Deep, Windows |

Planning for the next TouchXPRT

By Justin Greene

on August 29, 2019

We’re in the very early planning stages for the next version of TouchXPRT, and we’d love to hear any suggestions you may have. What do you like or dislike about TouchXPRT? What features do you hope to see in a new version?

For those who are unfamiliar with TouchXPRT, it’s a benchmark for evaluating the performance of Windows 10 devices. TouchXPRT 2016, the most recent version, runs tests based on five everyday scenarios (Beautify Photos, Blend Photos, Convert Videos for Sharing, Create Music Podcast, and Create Slideshow from Photos) and produces results for each of the five scenarios plus an overall score. The benchmark is available two ways: as a Universal Windows App in the Microsoft Store and as a sideload installer package on TouchXPRT.com.

When we begin work on a new version of any benchmark, one of the first steps we take is to assess its workloads to determine whether they will provide value during the years ahead. This step involves evaluating whether to update test content such as photos and videos to more contemporary file resolutions and sizes, and can also involve removing workloads or adding completely new ones. Should we keep the TouchXPRT workloads listed above or investigate other use cases? Should we research potential AI-related workloads? What do you think?

As we did with MobileXPRT 3 and HDXPRT 4 earlier this year, we’re also planning to update the TouchXPRT UI to improve the look of the benchmark and make it easier to use. We’re just at the beginning of this process, so any feedback you send has a chance to really shape the future of the benchmark.

On a related note, TouchXPRT 2016 testers who use the installer package available on TouchXPRT.com may have noticed that the package has a new file name (TX2016.6.52.0_8.19.19.zip). Microsoft requires developers to assign a security certificate to all sideload apps, and the new TouchXPRT file contains a refreshed certificate. We did not change the benchmark in any other way, so scores from this package are comparable to previous TouchXPRT 2016 scores.

Justin

Posted in Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, HDXPRT 4, MobileXPRT 3, Performance benchmarking, TouchXPRT, TouchXPRT 2016, What makes a good benchmark?, Windows, Windows 10 | Tagged benchmark, Microsoft, TouchXPRT, Windows, Windows 10 |

Navigating the AIXPRT Community Preview download page just got easier

By Justin Greene

on July 25, 2019

AIXPRT Community Preview 2 (CP2) has been generating quite a bit of interest among the BenchmarkXPRT Development Community and members of the tech press. We’re excited that the tool has piqued curiosity and that folks are recognizing its value for technical analysis. When talking with folks about test setup and configuration, we keep hearing the same questions:

How do I find the exact toolkit or package that I need?
How do I find the instructions for a specific toolkit?
What test configuration variables are most important for producing consistent, relevant results?
How do I know which values to choose when configuring options such as iterations, concurrent instances, and batch size?

In the coming weeks, we’ll be working to provide detailed answers to questions about test configuration. In response to the confusion about finding specific packages and instructions, we’ve redesigned the CP2 download page to make it easier for you to find what you need. Below, we show a snapshot from the new CP2 download table. Instead of having to download the entire CP2 package that includes the OpenVINO, TensorFlow, and TensorRT in TensorFlow test packages, you can now download one package at a time. In the Documentation column, we’ve posted package-specific instructions, so you won’t have to wade through the entire installation guide to find the instructions you need.

We hope these changes make it easier for people to experiment with AIXPRT. As always, please feel free to contact us with any questions or comments you may have.

Justin

Posted in AI, AIXPRT, AMD, benchmark, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Intel, Linux, NVIDIA, OpenVINO, TensorFlow, TensorRT, Windows 10 | Tagged AI, AIXPRT, Community Preview, CPU, GPU, Linux, machine learning, OpenVINO, TensorFlow, TensorRT |

Category: Collaborative benchmark development

Planning for the next CrXPRT

How to use alternate configuration files with AIXPRT

AIXPRT Community Preview 3 is here!

An update on AIXPRT development

Planning for the next TouchXPRT

Navigating the AIXPRT Community Preview download page just got easier

Check out the other XPRTs: