BenchmarkXPRT Blog banner

Category: Collaborative benchmark development

WebXPRT: What would you like to see?

Justin Greene
By Justin Greene
on July 11, 2019

At over 412,000 runs and counting, WebXPRT is our most popular benchmark. From the first release in 2013, it’s been popular with device manufacturers, developers, tech journalists, and consumers because it’s easy to run, it runs on almost anything with a web browser, and it evaluates device performance using the types of web-based tasks that people are likely to encounter on a daily basis.

With each new version of WebXPRT, we analyze browser development trends to make sure the test’s underlying web technologies and workload scenarios adequately reflect the ways people are using their browsers to work and play. BenchmarkXPRT Development Community members can play an important part in that process by sending us feedback on existing tests and suggestions for new workloads to include.

For example, when we released WebXPRT 3, we updated the photo workloads with new images and a deep learning task used for image classification. We also added an optical character recognition task in the Encrypt Notes and OCR scan workload, and combined part of the DNA Sequence Analysis scenario with a writing sample/spell check scenario to simulate online homework in an all-new Online Homework workload.

Consider for a moment what an ideal future version of WebXPRT would look like for you. Are there new web technologies or workload scenarios that you would like to see? Would you be interested in an associated battery life test? Should we include experimental tests? We’re interested in what you have to say, so please feel free to contact us with your thoughts or questions.

If you’re just now learning about WebXPRT, we offer several resources to help you better understand the benchmark and its range of uses. For a general overview of why WebXPRT matters, watch our video titled What is WebXPRT and why should I care? To read more about the details of the benchmark’s development and structure, check out the Exploring WebXPRT 3 white paper. To see WebXPRT 2015 and WebXPRT 3 scores from a wide range of processors, visit the WebXPRT 3 Processor Comparison Chart.

We look forward to hearing from you!

Justin

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, Future of performance evaluation, JavaScript, Performance benchmarking, Web-based testing, WebXPRT, WebXPRT 2013, WebXPRT 2015, WebXPRT 3, WebXPRT Processor Comparison Chart, What makes a good benchmark?, White papers | Tagged , , , , , |

The MobileXPRT 3 source code is now available

Justin Greene
By Justin Greene
on June 27, 2019

We’re excited to announce that the MobileXPRT 3 source code is now available to BenchmarkXPRT Development Community members!

Download the MobileXPRT 3 source here (login required).

We’ve also posted a download link on the MobileXPRT tab in the Members’ Area, where you will find instructions for setting up and configuring a local instance of MobileXPRT 3.

As part of our community model for software development, source code for each of the XPRTs is available to anyone who joins the community. If you’d like to review XPRT source code, but haven’t yet joined the community, we encourage you to join! Registration is quick and easy, and if you work for a company or organization with an interest in benchmarking, you can join the community for free. Simply fill out the form with your company e-mail address and select the option to be considered for a free membership. We’ll contact you to verify the address and then activate your membership.

If you have any other questions about community membership or XPRT source code, feel free to contact us. We look forward to hearing from you!

Justin

Posted in Android, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Mobile devices, MobileXPRT, MobileXPRT 3, Open development, Performance testing on tablets, Source code | Tagged , , , , , , |

Making AIXPRT easier to use

Justin Greene
By Justin Greene
on June 20, 2019

We’re glad to see so much interest in the AIXPRT CP2 build. Over the past few days, we’ve received two questions about the setup process: 1) where to find instructions for setting up AIXPRT on Windows, and 2) whether we could make it easier to install Intel OpenVINO on test systems.

In response to the first question, testers can find the relevant instructions for each framework in the readme files included in the AIXPRT install package. Instructions for Windows installation are in section 3 of the OpenVINO and TensorFlow readmes. Please note that whether you’re running AIXPRT on Ubuntu or Windows, be sure to read the “Known Issues” section in the readme, as there may be issues relevant to your specific configuration.

The readme files for each respective framework in the CP2 package are located here:

  • AIXPRT_0.5_CP2\AIXPRT_OpenVINO_0.5_CP2.zip\AIXPRT\Modules\Deep-Learning
  • AIXPRT_0.5_CP2\AIXPRT_TensorFLow_0.5_CP2.zip\AIXPRT\Modules\Deep-Learning
  • AIXPRT_0.5_CP2\AIXPRT_TensorFlow_TensorRT_0.5_CP2.zip\AIXPRT\Modules\Deep-Learning


We’re also working on consolidating the instructions into a central document that will make it easier for everyone to find the instructions they need.

In response to the question about OpenVINO installation, we’re working on an AIXPRT CP2 package that includes a precompiled version of OpenVINO R5.0.1 for easy installation on Windows via a few quick commands, and a script that installs the necessary OpenVINO dependencies. We’re currently testing the build, and we’ll make it available to testers as soon as possible.

The tests themselves will not change, so the new build will not influence existing results from Ubuntu or Windows. We hope it will simply facilitate the setup and testing process for many users.

We appreciate each bit of feedback that we receive, so if you have any suggestions for AIXPRT, please let us know!

Justin

Posted in AI, AIXPRT, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Linux, Machine learning, OpenVINO, TensorFlow, TensorRT, Ubuntu, What makes a good benchmark?, Windows 10 | Tagged , , , , , , , |

News on AIXPRT development

Justin Greene
By Justin Greene
on June 13, 2019

(more…)

Posted in AI, AIXPRT, Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Cross-platform benchmarks, Machine learning, MXNet, OpenVINO, TensorFlow, TensorRT, Wide and Deep | Tagged , , , , , , , |

Improvements to the AIXPRT results table

Justin Greene
By Justin Greene
on May 30, 2019

Over the last few weeks, we’ve gotten great feedback about the kinds of data points people are looking for in AIXPRT results, as well as suggestions for how to improve the AIXPRT results viewer. To make it easier for visitors to find what they’re looking for, we’ve made a number of changes:

  • You can now filter results in categories such as framework, target hardware, batch size, and precision, and can designate minimum throughput and maximum latency scores. When you select a value from a drop-down menu or enter text, the results change immediately to reflect the filter.
  • You can search for variables such as processor vendor or processor speed.
  • The viewer displays eight results per page by default and lets you change this to 16, 48, or Show all.

 

The following features of the viewer, which have been present previously, can help you to navigate more efficiently:

  • Click the tabs at the top of the table to switch from ResNet-50 network results to SSD-MobileNet network results.
  • Click the header of any column to sort the data on that variable. One click sorts A-Z and two clicks sort Z-A.
  • Click the link in the Source column to visit a detailed page on that result. The page contains additional test configuration and system hardware information and lets you download results files.

 

We hope these changes will improve the utility of the results table. We’ll continue to add features to improve the experience. If you have any suggestions, please let us know!

Justin

Posted in AI, AIXPRT, Benchmarking, Collaborative benchmark development, Community Preview, Cross-platform benchmarks, OpenVINO, Performance benchmarking, TensorFlow, TensorRT | Tagged , , , , |

We want to hear your thoughts about the AIXPRT development schedule

Justin Greene
By Justin Greene
on May 23, 2019

We released the second AIXPRT Community Preview (CP2) about two weeks ago. The main additions in CP2 were the ability to run certain test configurations in Windows (OpenVINO CPU/GPU and TensorFlow CPU), the option to download the installer package from the AIXPRT tab in the XPRT Members’ Area, and a demo mode.

We’re also investigating ways to support TensorFlow GPU and TensorFlow-TensorRT testing in Windows, and we’d like to eventually add support for TensorRT testing in Ubuntu and Windows. If development and pre-release testing go as planned, we may roll out some of these extra features by the end of June. However, it’s possible that getting all the pieces that we want in place will require a multi-step release process. If so, we’re considering two approaches: (1) issuing a third community preview (CP3) and (2) preparing a general availability (GA) release, to which we would add features over the months following the release. Neither of these paths is likely to affect test results from the currently supported configurations.

Would you like to work with another community preview, or would it be better for us to move straight to a GA release and add features as they become ready? We want to follow the approach that the majority of community members prefer, so please let us know what you think. As always, we also welcome any questions, concerns, or suggestions regarding the AIXPRT development process.

Justin

Posted in AI, AIXPRT, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Future of performance evaluation, OpenVINO, Performance benchmarking, TensorFlow, TensorRT, What makes a good benchmark?, Windows 10 | Tagged , , , , , , , |

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2025   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x