Category: What makes a good benchmark?

Thinking ahead to WebXPRT 4

on March 5, 2020

It’s been about two years since we released WebXPRT 3, and we’re starting to think about the WebXPRT 4 development cycle. With over 529,000 runs to date, WebXPRT continues to be our most popular benchmark because it’s quick and easy to run, it runs on almost anything with a web browser, and it evaluates performance using the types of web technologies that many people use every day.

For each new version of WebXPRT, we start the development process by looking at browser trends and analyzing the feasibility of incorporating new web technologies into our workload scenarios. For example, in WebXPRT 3, we updated the Organize Album workload to include an image-classification task that uses deep learning. We also added an optical character recognition task to the Encrypt Notes and OCR scan workload, and introduced a new Online Homework workload that combined part of the DNA Sequence Analysis scenario with a writing sample/spell check scenario.

Here are the current WebXPRT 3 workloads:

Photo Enhancement: Applies three effects, each using Canvas, to two photos.
Organize Album Using AI: Detects faces and classifies images using the ConvNetJS neural network library.
Stock Option Pricing: Calculates and displays graphic views of a stock portfolio using Canvas, SVG, and dygraphs.js.
Encrypt Notes and OCR Scan: Encrypts notes in local storage and scans a receipt using optical character recognition.
Sales Graphs: Calculates and displays multiple views of sales data using InfoVis and d3.js.
Online Homework: Performs science and English assignment tasks using Web Workers and Typo.js spell check.

What new technologies or workload scenarios should we add? Are there any existing features we should remove? Would you be interested in an associated battery life test? We want to hear your thoughts and ideas about WebXPRT, so please tell us what you think!

Justin

Posted in AI, benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, image classification, JavaScript, Performance benchmarking, WebXPRT, WebXPRT 3, What makes a good benchmark? | Tagged benchmark, browsers, WebXPRT, WebXPRT 3 |

An update on AIXPRT development

By Justin Greene

on September 26, 2019

It’s been a while since we last discussed the AIXPRT Community Preview 3 (CP3) release schedule, so we want to let everyone know where things stand. Testing for CP3 has taken longer than we predicted, but we believe we’re nearly ready for the release.

Testers can expect three significant changes in AIXPRT CP3. First, we updated support for the Ubuntu test packages. During the initial development phase of AIXPRT, Ubuntu version 16.04 LTS (Long Term Support) was the most current LTS version, but version 18.04 is now available.

Second, we have added TensorRT test packages for Windows and Ubuntu. Previously, AIXPRT testers could test only the TensorFlow variant of TensorRT. Now, they can use TensorRT to test systems with NVIDIA GPUs.

Third, we have added the Wide and Deep recommender system workload with the MXNet toolkit. Recommender systems are AI-based information-filtering tools that learn from end user input and behavior patterns and try to present them with optimized outputs that suit their needs and preferences. If you’ve used Netflix, YouTube, or Amazon accounts, you’ve encountered recommender systems that learn from your behavior.

Currently, the recommender system workload in AIXPRT CP3 is available for Ubuntu testing, but not for Windows. Recommender system inference workloads typically run on datacenter hardware, which tends to be Linux based. If enough community members are interested in running the MXNet/Wide and Deep test package on Windows, we can investigate what that would entail. If you’d like to see that option, please let us know.

As always, if you have any questions about the AIXPRT development process, feel free to ask!

Justin

Posted in AI, AIXPRT, benchmark, Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Linux, Machine learning, MXNet, Performance benchmarking, Ubuntu, What makes a good benchmark?, Wide and Deep, Windows |

Planning for the next TouchXPRT

By Justin Greene

on August 29, 2019

We’re in the very early planning stages for the next version of TouchXPRT, and we’d love to hear any suggestions you may have. What do you like or dislike about TouchXPRT? What features do you hope to see in a new version?

For those who are unfamiliar with TouchXPRT, it’s a benchmark for evaluating the performance of Windows 10 devices. TouchXPRT 2016, the most recent version, runs tests based on five everyday scenarios (Beautify Photos, Blend Photos, Convert Videos for Sharing, Create Music Podcast, and Create Slideshow from Photos) and produces results for each of the five scenarios plus an overall score. The benchmark is available two ways: as a Universal Windows App in the Microsoft Store and as a sideload installer package on TouchXPRT.com.

When we begin work on a new version of any benchmark, one of the first steps we take is to assess its workloads to determine whether they will provide value during the years ahead. This step involves evaluating whether to update test content such as photos and videos to more contemporary file resolutions and sizes, and can also involve removing workloads or adding completely new ones. Should we keep the TouchXPRT workloads listed above or investigate other use cases? Should we research potential AI-related workloads? What do you think?

As we did with MobileXPRT 3 and HDXPRT 4 earlier this year, we’re also planning to update the TouchXPRT UI to improve the look of the benchmark and make it easier to use. We’re just at the beginning of this process, so any feedback you send has a chance to really shape the future of the benchmark.

On a related note, TouchXPRT 2016 testers who use the installer package available on TouchXPRT.com may have noticed that the package has a new file name (TX2016.6.52.0_8.19.19.zip). Microsoft requires developers to assign a security certificate to all sideload apps, and the new TouchXPRT file contains a refreshed certificate. We did not change the benchmark in any other way, so scores from this package are comparable to previous TouchXPRT 2016 scores.

Justin

Posted in Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, HDXPRT 4, MobileXPRT 3, Performance benchmarking, TouchXPRT, TouchXPRT 2016, What makes a good benchmark?, Windows, Windows 10 | Tagged benchmark, Microsoft, TouchXPRT, Windows, Windows 10 |

WebXPRT: What would you like to see?

By Justin Greene

on July 11, 2019

At over 412,000 runs and counting, WebXPRT is our most popular benchmark. From the first release in 2013, it’s been popular with device manufacturers, developers, tech journalists, and consumers because it’s easy to run, it runs on almost anything with a web browser, and it evaluates device performance using the types of web-based tasks that people are likely to encounter on a daily basis.

With each new version of WebXPRT, we analyze browser development trends to make sure the test’s underlying web technologies and workload scenarios adequately reflect the ways people are using their browsers to work and play. BenchmarkXPRT Development Community members can play an important part in that process by sending us feedback on existing tests and suggestions for new workloads to include.

For example, when we released WebXPRT 3, we updated the photo workloads with new images and a deep learning task used for image classification. We also added an optical character recognition task in the Encrypt Notes and OCR scan workload, and combined part of the DNA Sequence Analysis scenario with a writing sample/spell check scenario to simulate online homework in an all-new Online Homework workload.

Consider for a moment what an ideal future version of WebXPRT would look like for you. Are there new web technologies or workload scenarios that you would like to see? Would you be interested in an associated battery life test? Should we include experimental tests? We’re interested in what you have to say, so please feel free to contact us with your thoughts or questions.

If you’re just now learning about WebXPRT, we offer several resources to help you better understand the benchmark and its range of uses. For a general overview of why WebXPRT matters, watch our video titled What is WebXPRT and why should I care? To read more about the details of the benchmark’s development and structure, check out the Exploring WebXPRT 3 white paper. To see WebXPRT 2015 and WebXPRT 3 scores from a wide range of processors, visit the WebXPRT 3 Processor Comparison Chart.

We look forward to hearing from you!

Justin

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, Future of performance evaluation, JavaScript, Performance benchmarking, Web-based testing, WebXPRT, WebXPRT 2013, WebXPRT 2015, WebXPRT 3, WebXPRT Processor Comparison Chart, What makes a good benchmark?, White papers | Tagged benchmark, browser, OCR, Performance, web, WebXPRT |

Making AIXPRT easier to use

By Justin Greene

on June 20, 2019

We’re glad to see so much interest in the AIXPRT CP2 build. Over the past few days, we’ve received two questions about the setup process: 1) where to find instructions for setting up AIXPRT on Windows, and 2) whether we could make it easier to install Intel OpenVINO on test systems.

In response to the first question, testers can find the relevant instructions for each framework in the readme files included in the AIXPRT install package. Instructions for Windows installation are in section 3 of the OpenVINO and TensorFlow readmes. Please note that whether you’re running AIXPRT on Ubuntu or Windows, be sure to read the “Known Issues” section in the readme, as there may be issues relevant to your specific configuration.

The readme files for each respective framework in the CP2 package are located here:

AIXPRT_0.5_CP2\AIXPRT_OpenVINO_0.5_CP2.zip\AIXPRT\Modules\Deep-Learning
AIXPRT_0.5_CP2\AIXPRT_TensorFLow_0.5_CP2.zip\AIXPRT\Modules\Deep-Learning
AIXPRT_0.5_CP2\AIXPRT_TensorFlow_TensorRT_0.5_CP2.zip\AIXPRT\Modules\Deep-Learning

We’re also working on consolidating the instructions into a central document that will make it easier for everyone to find the instructions they need.

In response to the question about OpenVINO installation, we’re working on an AIXPRT CP2 package that includes a precompiled version of OpenVINO R5.0.1 for easy installation on Windows via a few quick commands, and a script that installs the necessary OpenVINO dependencies. We’re currently testing the build, and we’ll make it available to testers as soon as possible.

The tests themselves will not change, so the new build will not influence existing results from Ubuntu or Windows. We hope it will simply facilitate the setup and testing process for many users.

We appreciate each bit of feedback that we receive, so if you have any suggestions for AIXPRT, please let us know!

Justin

Posted in AI, AIXPRT, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Linux, Machine learning, OpenVINO, TensorFlow, TensorRT, Ubuntu, What makes a good benchmark?, Windows 10 | Tagged AI, AIXPRT, machine learning, OpenVINO, TensorFlow, TensorRT, Ubuntu, Windows |

We want to hear your thoughts about the AIXPRT development schedule

By Justin Greene

on May 23, 2019

We released the second AIXPRT Community Preview (CP2) about two weeks ago. The main additions in CP2 were the ability to run certain test configurations in Windows (OpenVINO CPU/GPU and TensorFlow CPU), the option to download the installer package from the AIXPRT tab in the XPRT Members’ Area, and a demo mode.

We’re also investigating ways to support TensorFlow GPU and TensorFlow-TensorRT testing in Windows, and we’d like to eventually add support for TensorRT testing in Ubuntu and Windows. If development and pre-release testing go as planned, we may roll out some of these extra features by the end of June. However, it’s possible that getting all the pieces that we want in place will require a multi-step release process. If so, we’re considering two approaches: (1) issuing a third community preview (CP3) and (2) preparing a general availability (GA) release, to which we would add features over the months following the release. Neither of these paths is likely to affect test results from the currently supported configurations.

Would you like to work with another community preview, or would it be better for us to move straight to a GA release and add features as they become ready? We want to follow the approach that the majority of community members prefer, so please let us know what you think. As always, we also welcome any questions, concerns, or suggestions regarding the AIXPRT development process.

Justin

Posted in AI, AIXPRT, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Future of performance evaluation, OpenVINO, Performance benchmarking, TensorFlow, TensorRT, What makes a good benchmark?, Windows 10 | Tagged AIXPRT, Community Preview, CPU, GPU, OpenVINO, TensorFlow, TensorRT, Windows |

Category: What makes a good benchmark?

Thinking ahead to WebXPRT 4

An update on AIXPRT development

Planning for the next TouchXPRT

WebXPRT: What would you like to see?

Making AIXPRT easier to use

We want to hear your thoughts about the AIXPRT development schedule

Check out the other XPRTs: