BenchmarkXPRT Blog banner

Tag Archives: benchmark

Considering WebAssembly for WebXPRT 4

Justin Greene
By Justin Greene
on March 25, 2021

Earlier this month, we discussed a few of our ideas for possible changes in WebXPRT 4, including new web technologies that may work well in a browser benchmark. Today, we’re going to focus on one of those technologies, WebAssembly, in more detail.

WebAssembly (WASM) is a binary instruction format that works across all modern browsers. WASM provides a sandboxed environment that operates at native speeds and takes advantage of common hardware specs across platforms. WASM’s capabilities offer web developers a great deal of flexibility for running complex client applications in the browser. That level of flexibility may enable workload scenario options for WebXPRT 4 such as gaming, video editing, VR, virtual machines, and image recognition. We’re excited about those possibilities, but it remains to be seen which WASM use cases will meet the criteria we look for when considering new WebXPRT workloads, such as relevancy to real life, consistency and replicability, and the broadest-possible level of cross-browser support.

One WASM workload that we’re investigating is a web-based machine learning workload with TensorFlow for JavaScript (TensorFlow.js). TensorFlow.js offers pre-trained models for a wide variety of tasks, including image classification, object detection, sentence encoding, and natural language processing. TensorFlow.js originally used WebGL technology on the back end, but now it’s possible to run the workload using WASM. We could also use this technology to enhance one of WebXPRT’s existing AI-themed workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.

We’re can’t yet say that a WASM workload will definitely appear in WebXPRT 4, but the technology is promising. Do you have any experience with WASM, or ideas for WASM workloads? There’s still time for you to help shape the future of WebXPRT 4, so let us know what you think!

Justin

Posted in benchmark, BenchmarkXPRT, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, image processing, Machine learning, Performance benchmarking, TensorFlow, Virtual reality, Web-based testing, WebAssembly, WebGL, WebXPRT, WebXPRT 3 | Also tagged , , , , , , , ,

Improved CloudXPRT documentation is coming soon

Justin Greene
By Justin Greene
on January 21, 2021

CloudXPRT is undoubtedly the most complex tool in the XPRT family of benchmarks. To run the cloud-native benchmark’s multiple workloads across different hardware and software platforms, testers need two things: (1) at least a passing familiarity with a wide range of cloud-related toolkits, and (2) an understanding that changing even one test configuration variable can affect test results. While the complexity of CloudXPRT makes it a powerful and flexible tool for measuring application performance on real-world IaaS stacks, it also creates a steep learning curve for new users.

Benchmark setup and configuration can involve a number of complex steps, and the corresponding instructions should be thorough, unambiguous, and intuitive to follow. For all of the XPRT tools, we strive to publish documentation that provides quick, easy-to-find answers to the questions users might have. Community members have asked us to improve the clarity and readability of the CloudXPRT setup, configuration, and individual workload documentation. In response, we are working to create more—and better—CloudXPRT documentation.

If you’re intimidated by the benchmark’s complexity, helping you is one of our highest priorities. In the coming weeks and months, we’ll be evaluating all of our CloudXPRT documentation, particularly from the perspective of new users, and will release more information about the new documentation as it becomes available.

We also want to remind you of some of the existing CloudXPRT resources. We encourage everyone to check out the Introduction to CloudXPRT and Overview of the CloudXPRT Web Microservices Workload white papers. (Note that we’ll soon be publishing a paper on the benchmark’s data analytics workload.) Also, a couple of weeks ago, we published the CloudXPRT learning tool, which we designed to serve as an information hub for common CloudXPRT topics and questions, and to help tech journalists, OEM lab engineers, and everyone who is interested in CloudXPRT find the answers they need as quickly as possible.

Thanks to all who let us know that there was room for improvement in the CloudXPRT documentation. We rely on that kind of feedback and always welcome it. If you have any questions or suggestions regarding CloudXPRT or any of the other XPRTs, please let us know!

Justin

Posted in benchmark, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Cloud, CloudXPRT, Cross-platform benchmarks, Hosted cloud, Hybrid cloud, IaaS, microservices, on premises, Performance benchmarking, White papers | Also tagged , , , ,

The XPRTs in 2020: a year to remember

Justin Greene
By Justin Greene
on December 17, 2020

As 2020 comes to a close, we want to take this opportunity to review another productive year for the XPRTs. Readers of our newsletter are familiar with the stats and updates we include each month, but for our blog readers who don’t receive the newsletter, we’ve compiled some highlights below.

Benchmarks
In the past year, we released CrXPRT 2 and updated MobileXPRT 3 for testing on Android 11 phones. The biggest XPRT benchmark news was the release of CloudXPRT v1.0 and v1.01. CloudXPRT, our newest  benchmark, can accurately measure the performance of cloud applications deployed on modern infrastructure-as-a-service (IaaS) platforms, whether those platforms are paired with on-premises, private cloud, or public cloud deployments. 

XPRTs in the media
Journalists, advertisers, and analysts referenced the XPRTs thousands of times in 2020, and it’s always rewarding to know that the XPRTs have proven to be useful and reliable assessment tools for technology publications such as AnandTech, ArsTechnica, Computer Base, Gizmodo, HardwareZone, Laptop Mag, Legit Reviews, Notebookcheck, PCMag, PCWorld, Popular Science, TechPowerUp, Tom’s Hardware, VentureBeat, and ZDNet.

Downloads and confirmed runs
So far in 2020, we’ve had more than 24,200 benchmark downloads and 164,600 confirmed runs. Our most popular benchmark, WebXPRT, just passed 675,000 runs since its debut in 2013! WebXPRT continues to be a go-to, industry-standard performance benchmark for OEM labs, vendors, and leading tech press outlets around the globe.

Media, publications, and interactive tools
Part of our mission with the XPRTs is to produce materials that help testers better understand the ins and outs of benchmarking in general and the XPRTs in particular. To help achieve this goal, we’ve published the following in 2020:

We’re thankful for everyone who has used the XPRTs, joined the community, and sent questions and suggestions throughout 2020. This will be our last blog post of the year, but there’s much more to come in 2021. Stay tuned in early January for updates!

Justin

Posted in AIXPRT, AnandTech, Android 11, benchmark, Benchmarking, BenchmarkXPRT, Cloud, CloudXPRT, CrXPRT, IaaS, Interactive tools, MobileXPRT, MobileXPRT 3, Performance benchmarking, Phones, WebXPRT | Also tagged , , , , , , , , ,

The AIXPRT learning tool is now live (and a CloudXPRT version is on the way)!

Justin Greene
By Justin Greene
on December 10, 2020

We’re happy to announce that the AIXPRT learning tool is now live! We designed the tool to serve as an information hub for common AIXPRT topics and questions, and to help tech journalists, OEM lab engineers, and everyone who is interested in AIXPRT find the answers they need in as little time as possible.

The tool features four primary areas of content:

  • The Q&A section provides quick answers to the questions we receive most from testers and the tech press.
  • The AIXPRT: the basics section describes specific topics such as the benchmark’s toolkits, networks, workloads, and hardware and software requirements.
  • The testing and results section covers the testing process, metrics, and how to publish results.
  • The AI/ML primer provides brief, easy-to-understand definitions of key AI and ML terms and concepts for those who want to learn more about the subject.

The first screenshot below shows the home screen. To show how some of the popup information sections appear, the second screenshot shows the Inference tasks (workloads) entry in the AI/ML Primer section. 

We’re excited about the new AIXPRT learning tool, and we’re also happy to report that we’re working on a version of the tool for CloudXPRT. We hope to make the CloudXPRT tool available early next year, and we’ll post more information in the blog as we get closer to taking it live.

If you have any questions about the tool, please let us know!

Justin

Posted in AI, AIXPRT, benchmark, BenchmarkXPRT, Cloud, CloudXPRT, computer vision, Cross-platform benchmarks, data analytics, image classification, Machine learning, object detection, OpenVINO, Performance benchmarking, TensorFlow, TensorRT | Also tagged , , , , ,

Thinking ahead to the next HDXPRT

Justin Greene
By Justin Greene
on November 12, 2020

We’re currently formulating our 2021 development roadmap for the XPRTs. In addition to planning CloudXPRT and WebXPRT updates, we’re discussing the possibility of releasing HDXPRT 5 in 2021. It’s hard for me to believe, but it’s been about two and a half years since we started work on HDXPRT 4, and February 2021 will mark two years since the first HDXPRT 4 release. Windows PCs are more powerful than ever, so it’s a good time to talk about how we can enhance the benchmark’s ability to measure how well the latest systems handle real-world media technologies and applications.

When we plan a new version of an XPRT benchmark, one of our first steps is updating the benchmark’s workloads so that they will remain relevant in years to come. We almost always update application content, such as photos and videos, to contemporary file resolutions and sizes. For example, we added both higher-resolution photos and a 4K video conversion task in HDXPRT 4. Are there specific types of media files that you think would be especially relevant to high-performance media tasks over the next few years?

Next, we will assess the suitability of the real-world trial applications that the editing photos, editing music, and converting videos test scenarios use. Currently, these are Adobe Photoshop Elements, Audacity, CyberLink MediaEspresso, and HandBrake. Can you think of other applications that belong in a high-performance media processing benchmark?

In HDXPRT 4, we gave testers the option to target a system’s discrete graphics card during the video conversion workload. Has this proven useful in your testing? Do you have suggestions for new graphics-oriented workloads?

We’ll also strive to make the UI more intuitive, to simplify installation, and to reduce the size of the installation package. What elements of the current UI do you find especially useful or think we could improve? 

We welcome your answers to these questions and any additional suggestions or comments on HDXPRT 5. Send them our way!

Justin

Posted in Adobe, Application-based benchmarks, benchmark, BenchmarkXPRT, CloudXPRT, Collaborative benchmark development, Future of performance evaluation, HDXPRT, HDXPRT 4, HDXPRT development process, HDXPRT metrics, HDXPRT release cycle, Performance benchmarking, WebXPRT, What makes a good benchmark? | Also tagged , , , , , , ,

We’re working on an AIXPRT learning tool

Justin Greene
By Justin Greene
on September 10, 2020

For anyone interested in learning more about AIXPRT, the Introduction to AIXPRT white paper provides detailed information about its toolkits, workloads, system requirements, installation, test parameters, and results. However, for AIXPRT.com visitors who want to find the answers to specific AIXPRT-related questions quickly, a white paper can be daunting.

Because we want tech journalists, OEM lab engineers, and everyone who is interested in AIXPRT to be able to find the answers they need in as little time as possible, we’ve decided to develop a new learning tool that will serve as an information hub for common AIXPRT topics and questions.

The new learning tool will be available online through our site. It will offer quick bites of information about the fundamentals of AIXPRT, why the benchmark matters, the benefits of AIXPRT testing and results, machine learning concepts, key terms, and practical testing concerns.

We’re still working on the tool’s content and design. Because we’re designing this tool for you, we’d love to hear the topics and questions you think we should include. If you have any suggestions, please let us know!

Justin

Posted in AI, AIXPRT, benchmark, Benchmarking, Cross-platform benchmarks, Interactive tools, Machine learning, White papers | Also tagged , , ,

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2024   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x