BenchmarkXPRT Blog banner

Category: Benchmarking

The WebXPRT 4 results calculation white paper is now available

Justin Greene
By Justin Greene
on September 8, 2022

Last week, we published the Exploring WebXPRT 4 white paper. The paper describes the design and structure of WebXPRT 4, including detailed information about the benchmark’s harness, HTML5 and WebAssembly capability checks, and the structure of the performance test workloads. This week, to help WebXPRT 4 testers understand how the benchmark calculates results, we’ve published the WebXPRT 4 results calculation and confidence interval white paper.

The white paper explains the WebXPRT 4 confidence interval and how it differs from typical benchmark variability, and the formulas the benchmark uses to calculate the individual workload scenario scores and overall score. The paper also provides an overview of the statistical techniques WebXPRT uses to translate raw timings into scores.

To supplement the white paper’s discussion of the results calculation process, we’ve also published a results calculation spreadsheet that shows the raw data from a sample test run and reproduces the calculations WebXPRT uses to produce workload scores and the overall score.

The paper is available on WebXPRT.com and on our XPRT white papers page. If you have any questions about the WebXPRT results calculation process, please let us know!

Justin

Posted in benchmark, Browser-based benchmarks, HTML5, Performance benchmarking, results, WebAssembly, WebXPRT, WebXPRT 4, White papers |

The Exploring WebXPRT 4 white paper is now available

Justin Greene
By Justin Greene
on September 1, 2022

This week, we published the Exploring WebXPRT 4 white paper. It describes the design and structure of WebXPRT 4, including detailed information about the benchmark’s harness, HTML5 and WebAssembly (WASM) capability checks, and changes we’ve made to the structure of the performance test workloads. We explain the benchmark’s scoring methodology, how to automate tests, and how to submit results for publication. The white paper also includes information about the third-party functions and libraries that WebXPRT 4 uses during the HTML5 and WASM capability checks and performance workloads.

The Exploring WebXPRT 4 white paper promotes the high level of transparency and disclosure that is a core value of the BenchmarkXPRT Development Community. We’ve always believed that transparency builds trust, and trust is essential for a healthy benchmarking community. That’s why we involve community members in the benchmark development process and disclose how we build our benchmarks and how they work.

You can find the paper on WebXPRT.com and our XPRT white papers page. If you have any questions about WebXPRT 4, please let us know, and be sure to check out our other XPRT white papers.

Justin

Posted in Automation, benchmark, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Performance benchmarking, results, results submission, WebAssembly, WebXPRT, WebXPRT 3, WebXPRT 4, White papers | Tagged , , , , , , , , , |

XPRT possibilities with ChromeOS Flex

Justin Greene
By Justin Greene
on August 25, 2022

Recently, Tom’s Guide published an interesting article about how they used ChromeOS Flex to turn a ten-year-old Apple MacBook Pro into a functioning Chromebook by replacing the laptop’s macOS operating system with ChromeOS. ChromeOS Flex is a free Google tool that allows users to create a bootable USB drive that they can then use to install ChromeOS on a wide variety of hardware platforms that traditionally run other operating systems such as macOS or Windows. Because ChromeOS is a cloud-first, relatively low-overhead operating system, the ChromeOS Flex option could breathe new life into an old laptop that you have lying around.

Never having encountered a MacBook Pro with ChromeOS, we were interested to learn about Tom’s experience running XPRT benchmarks in this new environment. WebXPRT 4, WebXPRT 3, and the CrXPRT 2 performance test apparently ran without any issues, but we have not yet seen a CrXPRT 2 battery life result from a ChromeOS Flex environment. We plan to experiment with this soon.

We were happy to publish the results on our site, and will consider any ChromeOS Flex results we receive for publication. If you submit results from ChromeOS Flex testing, we ask that you use the “Additional information” field in the results submission form to clarify that you ran the tests in a ChromeOS Flex environment. This will prevent any possible confusion when we see a submission that lists a traditional macOS or Windows hardware platform along with a ChromeOS version number.

Do you have experience running CrXPRT or WebXPRT with ChromeOS Flex? We’d love to hear about it!

Justin

Posted in Apple, Battery life, Chrome OS, Chromebooks, CrXPRT, Google Chrome, Laptops, Performance benchmarking, results, results submission, WebXPRT, WebXPRT 4, Windows | Tagged , , , , , , , , , |

CloudXPRT status and next steps

Justin Greene
By Justin Greene
on August 11, 2022

We developed our first cloud benchmark, CloudXPRT, to measure the performance of cloud applications deployed on modern infrastructure as a service (IaaS) platforms. When we first released CloudXPRT in February of 2021, the benchmark included two test packages: a web microservices workload and a data analytics workload. Both supported on-premises and cloud service provider (CSP) testing with Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. 

CloudXPRT is our most complex benchmark, requiring sustained compatibility between many software components across multiple independent test environments. As vendors roll out updates for some components and stop supporting others, it’s inevitable that something will break. Since CloudXPRT’s launch, we’ve become aware of installation failures while attempting to set up CloudXPRT on Ubuntu virtual machines with GCP and Microsoft Azure. Additionally, while the web microservices workload continues to run in most instances with a few configuration tweaks and workarounds, the data analytics workload fails consistently due to compatibility issues with Minio, Prometheus, and Kafka within the Kubernetes environment. 

In response, we’re working to fix problems with the web microservices workload and bring all necessary components up to date. We’re developing an updated test package that will work on Ubuntu 22.04, using Kubernetes v1.23.7 and Kubespray v2.18.1. We’re also updating Kubernetes Metrics Server from v1beta1 to v1, and will incorporate some minor script changes. Our goal is to ensure successful installation and testing with the on-premises and CSP platforms that we supported when we first launched CloudXPRT.

We are currently focusing on the web microservices workload for two reasons. First, more users have downloaded it than the data analytics workload. Second, we think we have a clear path to success. Our plan is to publish the updated web microservices test package, and see what feedback and interest we receive from users about a possible data analytics refresh. The existing data analytics workload will remain available via CloudXPRT.com for the time being to serve as a reference resource.

We apologize for the inconvenience that these issues have caused. We’ll provide more information about a release timeline and final test package details here in the blog as we get closer to publication. If you have any questions about the future of CloudXPRT, please feel free to contact us!

Justin

Posted in Amazon Web Services, benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, Cloud, CloudXPRT, Cross-platform benchmarks, CSP, data analytics, Datacenter, Google Cloud Platform, Hosted cloud, Hybrid cloud, IaaS, Kubernetes, microservices, Microsoft Azure, On-premise, Performance benchmarking, web microservices | Tagged , , , , , , , , , , |

The XPRTs can help with your back-to-school shopping

Justin Greene
By Justin Greene
on August 4, 2022

The new school year is upon us, and learners of all ages are looking for tech devices that have the capabilities they will need in the coming year. The tech marketplace can be confusing, and competing claims can be hard to navigate. The XPRTs are here to help! Whether you’re shopping for a new phone, tablet, Chromebook, laptop, or desktop, the XPRTs can provide reliable, industry-trusted performance scores that can cut through all the noise.

A good place to start looking for scores is the WebXPRT 4 results viewer. The viewer displays WebXPRT 4 scores from over 175 devices—including many hot new releases—and we’re adding new scores all the time. To learn more about the viewer’s capabilities and how you can use it to compare devices, check out this blog post.

Another resource we offer is the XPRT results browser. The browser is the most efficient way to access the XPRT results database, which currently holds more than 3,000 test results from over 120 sources, including major tech review publications around the world, OEMs, and independent testers. It offers a wealth of current and historical performance data across all of the XPRT benchmarks and hundreds of devices. You can read more about how to use the results browser here.

Also, if you’re considering a popular device, chances are good that a recent tech review includes an XPRT score for that device. Two quick ways to find these reviews: (1) go to your favorite tech review site and search for “XPRT” and (2) go to a search engine and enter the device name and XPRT name (e.g., “Apple MacBook Air” and “WebXPRT”). Here are a few recent tech reviews that use one of the XPRTs to evaluate a popular device:

The XPRTs can help consumers make better-informed and more confident tech purchases. As this school year begins, we hope you’ll find the data you need on our site or in an XPRT-related tech review. If you have any questions about the XPRTs, XPRT scores, or the results database please feel free to ask!

Justin

Posted in Apple, benchmark, Benchmark metrics, Browser-based benchmarks, Chromebooks, CrXPRT, Education, Laptops, Mobile devices, Performance benchmarking, Phones, results, Tablets, WebXPRT, WebXPRT 3, WebXPRT 4 | Tagged , , , , , , , , , , , , |

We’ve moved WebXPRT Singapore to a new hosting environment

Justin Greene
By Justin Greene
on July 28, 2022

When we first released WebXPRT 2013, some users in mainland China reported slow download times when running the benchmark. In response, we set up a mirror host site in Singapore to facilitate WebXPRT testing in China and other East Asian countries. We continued this practice with subsequent WebXPRT versions, and currently offer Singapore-based instances of WebXPRT 4, WebXPRT 3, and WebXPRT 2015.

Until this past month, we used an Amazon Web Services (AWS) EC2-Classic environment to host the Singapore mirror site. Because Amazon retired the EC2-Classic environment, we had to migrate each of the WebXPRT Singapore instances to a new AWS Virtual Private Cloud (VPC) environment.  

We do not expect the new environment to affect WebXPRT Singapore testing or results, and have not yet observed any significant differences in WebXPRT performance scores while testing on the new site. If you have a different experience when testing on the new site or encounter interruptions when trying to access the test, please let us know!

Justin

Posted in Amazon Web Services, benchmark, Benchmarking, Browser-based benchmarks, Performance benchmarking, WebXPRT, WebXPRT 2013, WebXPRT 2015, WebXPRT 3, WebXPRT 4 |

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2024   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x