BenchmarkXPRT Blog banner

Tag Archives: benchmark

A huge milestone for XPRT runs and downloads!

Justin Greene
By Justin Greene
on July 1, 2021

We’re excited to have recently passed an important milestone: one million XPRT runs and downloads! Most importantly, that huge number does not just reflect past successes. As the chart below illustrates, XPRT use has grown steadily over the years. In 2021, we record, on average, more XPRT runs in one month (23,395) than we recorded in the entire first year we started tracking these stats (17,051).

We reached one million runs and downloads in about seven and a half years. At the current rate, we’ll reach two million in roughly three and a half more years. With WebXPRT 4 on the way, there’s a good chance we can reach that mark even sooner!

As always, we’re grateful for all the testers that have helped us reach this milestone. If you have any questions or comments about using any of the XPRTs to test your gear, let us know!

Justin

Posted in benchmark, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, History of benchmarking, WebXPRT, webxprt 4 | Also tagged , , , ,

Publishing CloudXPRT results from testing on pre-production gear

Justin Greene
By Justin Greene
on May 27, 2021

We recently received questions about whether we accept CloudXPRT results submissions from testing on pre-production gear, and how we would handle any differences between results from pre-production and production-level tests.  

To answer first question, we are not opposed to pre-production results submissions. We realize that vendors often want to include benchmark results in launch-oriented marketing materials they release before their hardware or software is publicly available. To help them do so, we’re happy to consider pre-production submissions on a case-by-case basis. All such submissions must follow the normal CloudXPRT results submission process, and undergo vetting by the CloudXPRT Results Review Group according to the standard review and publication schedule. If we decide to publish pre-production results on our site, we will clearly note their pre-production status.

In response to the second question, the CloudXPRT Results Review Group will handle any challenges to published results or perceived discrepancies between pre-production and production-level results on a case-by-case basis. We do not currently have a formal process for challenges; anyone who would like to initiate a challenge or express comments or concerns about a result should address the review group via benchmarkxprtsupport@principledtechnologies.com. Our primary concern is always to ensure that published results accurately reflect the performance characteristics of production-level hardware and software. If it becomes necessary to develop more policies in the future, we’ll do so, but we want to keep things as simple as possible.

If you have any questions about the CloudXPRT results submission process, please let us know!

Justin

Posted in benchmark, Benchmark metrics, BenchmarkXPRT, Cloud, CloudXPRT, Datacenter, results, results submission, Server | Also tagged , , , ,

CrXPRT 2 battery life error on Chrome 89 and 90

Justin Greene
By Justin Greene
on May 20, 2021

In recent lab tests, we’ve encountered an error during the CrXPRT 2 battery life test that prevents the test from completing and producing a battery life estimate. As the screenshot below shows, when the error occurs, CrXPRT stops running its normal workload cycle and produces a “Test Error” page. We have seen this behavior on systems running Chrome OS v89.x and v90.x, across multiple vendor platforms. In our testing, Chromebooks running  Chrome OS v88.x and earlier versions continue to complete the battery life test without any issues.

The error occurs consistently on every Chromebook running v89.x or v90.x that we’ve tested so far. However, the timing of the error varies from run to run on the same system. Sometimes, CrXPRT stops running after only a few workload iterations, while at other times, the battery life test runs almost to completion before producing the error.

We’re actively investigating this problem, but have not yet identified the root cause. We apologize for the inconvenience that this error may be causing CrXPRT 2 testers. As soon as we identify the root cause of the problem and have ideas about possible solutions, we will share that information here in the blog. If you have any insight into recent Chrome OS changes or flag settings that could be causing this problem, please let us know!

Justin

Posted in battery life, benchmark, Chrome OS, Chromebooks, CrXPRT, Google Chrome | Also tagged , , , , ,

The WebXPRT 4 tech press feedback survey

Justin Greene
By Justin Greene
on May 6, 2021

Device reviews in publications such as AnandTech, Notebookcheck, and PCMag, among many others, often feature WebXPRT test results, and we appreciate the many members of the tech press that use WebXPRT. As we move forward with the WebXPRT 4 development process, we’re especially interested in learning what longtime users would like to see in a new version of the benchmark.  

In previous posts, we’ve asked people to weigh in on the potential addition of a WebAssembly workload or a battery life test. We’d also like to ask experienced testers some other test-related questions. To that end, this week we’ll be sending a WebXPRT 4 survey directly to members of the tech press who frequently publish WebXPRT test results.

Regardless of whether you are a member of the tech press, we invite you to participate by sending your answers to any or all the questions below to benchmarkxprtsupport@principledtechnologies.com. We ask you to do so by the end of May.

  • Do you think WebXPRT 3’s selection of workload scenarios is representative of modern web tasks?
  • How do you think WebXPRT compares to other common browser-based benchmarks, such as JetStream, Speedometer, and Octane?
  • Are there web technologies that you’d like us to include in additional workloads?
  • Are you happy with the WebXPRT 3 user interface? If not, what UI changes would you like to see?
  • Are there any aspects of WebXPRT 2015 that we changed in WebXPRT 3 that you’d like to see us change back?
  • Have you ever experienced significant connection issues when testing with WebXPRT?
  • Given the array of workloads, do you think the WebXPRT runtime is reasonable? Would you mind if the average runtime were a bit longer?
  • Are there any other aspects of WebXPRT 3 that you’d like to see us change?

If you’d like to discuss any topics that we did not cover in the questions above, please feel free to include additional comments in your response. We look forward to hearing your thoughts!

Justin

Posted in Battery life, benchmark, Benchmarking, BenchmarkXPRT, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, Future of performance evaluation, Performance benchmarking, WebAssembly, WebXPRT, WebXPRT 2015, WebXPRT 3, webxprt 4, What makes a good benchmark? | Also tagged , , , , , , , , , ,

WebXPRT passes the 750,000-run milestone!

Justin Greene
By Justin Greene
on April 29, 2021

We’re excited to see that users have successfully completed over 750,000 WebXPRT runs! If you’ve run WebXPRT in any of the more than 654 cities and 68 countries from which we’ve received complete test data—including newcomers Belize, Cambodia, Croatia, and Pakistan—we’re grateful for your help. We could not have reached this milestone without you!

As the chart below illustrates, WebXPRT use has grown steadily over the years. We now record, on average, almost twice as many WebXPRT runs in one month as we recorded in the entirety of our first year. In addition, with over 82,000 runs to date in 2021, there are no signs that growth is slowing.

Developing a new benchmark is never easy, and the obstacles multiply when you attempt to create a cross-platform benchmark, such as WebXPRT, that will run on a wide variety of devices. Establishing trust with the benchmarking community is another challenge. Transparency, consistency, and technical competency on our part are critical factors in building that trust, but the people who take time out of their busy schedules to run the benchmark for the first time also play a role. We thank all of the manufacturers, OEM labs, and members of the tech press who decided to give WebXPRT a try, and we look forward to your input as we continue to improve WebXPRT in the years to come. 

If you have any questions or comments about WebXPRT, we’d love to hear from you!

Justin

Posted in benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, Browser-based benchmarks, Performance benchmarking, WebXPRT, WebXPRT 2013, WebXPRT 3, webxprt 4, What makes a good benchmark? | Also tagged , , , , ,

Considering WebAssembly for WebXPRT 4

Justin Greene
By Justin Greene
on March 25, 2021

Earlier this month, we discussed a few of our ideas for possible changes in WebXPRT 4, including new web technologies that may work well in a browser benchmark. Today, we’re going to focus on one of those technologies, WebAssembly, in more detail.

WebAssembly (WASM) is a binary instruction format that works across all modern browsers. WASM provides a sandboxed environment that operates at native speeds and takes advantage of common hardware specs across platforms. WASM’s capabilities offer web developers a great deal of flexibility for running complex client applications in the browser. That level of flexibility may enable workload scenario options for WebXPRT 4 such as gaming, video editing, VR, virtual machines, and image recognition. We’re excited about those possibilities, but it remains to be seen which WASM use cases will meet the criteria we look for when considering new WebXPRT workloads, such as relevancy to real life, consistency and replicability, and the broadest-possible level of cross-browser support.

One WASM workload that we’re investigating is a web-based machine learning workload with TensorFlow for JavaScript (TensorFlow.js). TensorFlow.js offers pre-trained models for a wide variety of tasks, including image classification, object detection, sentence encoding, and natural language processing. TensorFlow.js originally used WebGL technology on the back end, but now it’s possible to run the workload using WASM. We could also use this technology to enhance one of WebXPRT’s existing AI-themed workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.

We’re can’t yet say that a WASM workload will definitely appear in WebXPRT 4, but the technology is promising. Do you have any experience with WASM, or ideas for WASM workloads? There’s still time for you to help shape the future of WebXPRT 4, so let us know what you think!

Justin

Posted in benchmark, BenchmarkXPRT, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, image processing, Machine learning, Performance benchmarking, TensorFlow, Virtual reality, Web-based testing, WebAssembly, WebGL, WebXPRT, WebXPRT 3 | Also tagged , , , , , , , ,

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2025   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x