BenchmarkXPRT Blog banner

Category: Performance benchmarking

Using WebXPRT 3 to compare the performance of popular browsers (Round 2)

It’s been nine months since we’ve published a WebXPRT 3 browser performance comparison, so we decided to put the newest versions of popular browsers through the paces to see if the performance rankings have changed since our last round of tests.

We used the same laptop as last time: a Dell XPS 13 7930 with an Intel Core i3-10110U processor and 4 GB of RAM running Windows 10 Home, updated to version 1909 (18363.1139). We installed all current Windows updates and tested on a clean system image. After the update process completed, we turned off updates to prevent them from interfering with test runs. We ran WebXPRT 3 three times on five browsers: Brave, Google Chrome, Microsoft Edge, Mozilla Firefox, and Opera. The posted score for each browser is the median of the three test runs.

In our last round of tests, the four Chromium-based browsers (Brave, Chrome, Edge, and Opera) produced scores that were nearly identical. Only Mozilla Firefox produced a significantly different (and better) score. The parity of the Chromium-based browsers was not surprising, considering they have the same underlying foundation.

In this round of testing, the Chromium-based browsers again produced very close scores, although Brave’s performance lagged by about 4 percent. Firefox again separated itself from the pack with a higher score. With the exception of Chrome, which produced an identical score as last time, every browser’s score was slightly slower than before. There are many possible reasons for this, including increased overhead in the browsers or changes in Windows, and the respective slowdowns for each browser will probably be unnoticeable to most users during everyday tasks.

Do these results mean that Mozilla Firefox will provide you with a speedier web experience? As we noted in the last comparison, a device with a higher WebXPRT score will probably feel faster during daily use than one with a lower score. For comparisons on the same system, however, the answer depends in part on the types of things you do on the web, how the extensions you’ve installed affect performance, how frequently the browsers issue updates and incorporate new web technologies, and how accurately each browsers’ default installation settings reflect how you would set up that browser for your daily workflow.

In addition, browser speed can increase or decrease significantly after an update, only to swing back in the other direction shortly thereafter. OS-specific optimizations can also affect performance, such as with Edge on Windows 10 and Chrome on Chrome OS. All of these variables are important to keep in mind when considering how browser performance comparison results translate to your everyday experience.

What are your thoughts on browser performance? Let us know!

Justin

A first look at the upcoming AIXPRT learning tool

Last month, we announced that we’re working on a new AIXPRT learning tool. Because we want tech journalists, OEM lab engineers, and everyone who is interested in AIXPRT to be able to find the answers they need in as little time as possible, we’re designing this tool to serve as an information hub for common AIXPRT topics and questions.

We’re still finalizing aspects of the tool’s content and design, so some details may change, but we can now share a sneak peak of the main landing page. In the screenshot below, you can see that the tool will feature four primary areas of content:

  • The FAQ section will provide quick answers to the questions we receive most from testers and the tech press.
  • The AIXPRT basics section will describe specific topics such as the benchmark’s toolkits, networks, workloads, and hardware and software requirements.
  • The testing and results section will cover the testing process, the metrics the benchmark produces, and how to publish results.
  • The AI/ML primer will provide brief, easy-to-understand definitions of key AI and ML terms and concepts for those who want to learn more about the subject.

We’re excited about the new AIXPRT learning tool, and will share more information here in the blog as we get closer to a release date. If you have any questions about the tool, please let us know!

Justin

New MobileXPRT 3 installations may crash on Android 11

We recently received a tech support inquiry about problems with new MobileXPRT 3 installations on some Android 11 phones. The tester installed MobileXPRT 3 on a selection of phones running Android 11, and the app crashed immediately upon opening. We were able to reproduce the issue on multiple phones in our lab, and currently know that the issue may happen on the Google Pixel 3, Google Pixel 4a 5G, Google Pixel 4XL, Google Pixel 5, and the OnePlus 8T (running Android 11 with an Oxygen OS skin).

MobileXPRT 3 continues to run without issues on Android 9 and 10 phones. When we updated an Android 10 phone with an existing MobileXPRT 3 installation to Android 11, we found that the benchmark ran successfully. This suggests a lack of fundamental incompatibilities between MobileXPRT 3 and current versions of Android 11. Because some of our lab techs experienced crashes immediately after the app asked for permissions, we think it’s possible that new permissions-setting requirements in Android 11 are causing the problem.

We’re currently working to isolate the problem and identify a course of action. We’ll share more information here in the blog as soon as possible. If you’ve encountered this problem in your testing, we apologize for the inconvenience, and we’re thankful for your patience as we work towards a solution.

If you have any information you’d like to share about running MobileXPRT 3 on Android 11, please let us know!

Justin

Next up: a white paper about the CloudXPRT data analytics workload

Soon, we’ll be publishing a CloudXPRT white paper that focuses on the benchmark’s data analytics workload. We summarized the workload in the Introduction to CloudXPRT white paper, but in the same way that the Overview of the CloudXPRT Web Microservices Workload paper did, the new paper will discuss the workload in much greater detail.

In addition to providing practical information about the installation package and minimum system requirements for the data analytics workload, the paper will describe test configuration variables, structural components, task workflows, and test metrics. It will also include guidance on interpreting test results and submitting them for publication.

As we’ve noted, CloudXPRT is one of the more complex tools in the XPRT family, with no shortage of topics to explore. Possible future topics include the impact of adjusting specific test configuration options, recommendations for results reporting, and methods for results analysis. If there are specific topics that you’d like us to address in future white papers, please feel free to send us your ideas!

We hope that the upcoming Overview of the CloudXPRT Data Analytics Workload paper will serve as a go-to resource for CloudXPRT testers, and will answer any questions you have about the workload. Once it goes live, we’ll provide links in the Helpful Info box on CloudXPRT.com and the CloudXPRT section of our XPRT white papers page.

If you have any questions, please let us know!

Justin

Following up

This week, we’re sharing news on two topics that we’ve discussed here in the blog over the past several months: CloudXPRT v1.01 and a potential AIXPRT OpenVINO update.

CloudXPRT v1.01

Last week, we announced that we were very close to releasing an updated CloudXPRT build (v1.01) with two minor bug fixes, an improved post-test results processing script, and an adjustment to one of our test configuration recommendations. Our testing and prep is complete, and the new version is live in the CloudXPRT GitHub repository and on our site!

None of the v1.01 changes affect performance or test results, so scores from the new build are comparable to those from previous CloudXPRT builds. If you’d like to know more about the changes, take a look at last week’s blog post.

The AIXPRT OpenVINO update

In late July, we discussed our plans to update the AIXPRT OpenVINO packages with OpenVINO 2020.3 Long-Term Support (LTS). While there are no known problems with the existing AIXPRT OpenVINO package, the LTS version targets environments that benefit from maximum stability and don’t require a constant stream of new tools and feature changes, so we thought it would be well suited for a benchmark like AIXPRT.

We initially believed that the update process would be relatively simple, and we’d be able to release a new AIXPRT OpenVINO package in September. However, we’ve discovered that the process is involved enough to require substantial low-level recoding. At this time, it’s difficult to estimate when the updated build will be ready for release. For any testers looking forward to the update, we apologize for the delay.

If you have any questions or comments about these or any other XPRT-related topics, please let us know!

Justin

Fixes for minor CloudXPRT bugs are on the way

We want to let CloudXPRT testers know that we’re close to releasing an updated version (build 1.01) with two minor bug fixes, an improved post-test results processing script, and an adjustment to one of our test configuration recommendations. None of these changes will affect performance or test results, so scores from previous CloudXPRT builds will be comparable to those from the new build.

The most significant changes in CloudXPRT build 1.01 are as follows:

  • In previous builds, some testers encountered warnings during setup to update the version of Kubernetes Operations (kops) when testing on public-cloud platforms (the CloudXPRT 1.00 recommendation is kops version 1.16.0). We are adjusing the kops installation instructions in the setup instructions for the web microservices and data analytics workloads to prevent these warnings.
  • In previous builds, post-test cleanup instructions for public-cloud testing environments do not always delete all of the resources that CloudXPRT creates during setup. We are updating instructions to ensure a more thorough cleanup process. This change applies to test instructions for the web microservices and data analytics workloads.
  • We are reformatting the optional results graphs the web microservices postprocess program creates to make them easier to interpret.
  • In previous builds, the recommended time interval for the web-microservices workload is 120 seconds if the hpamode option is enabled and 60 seconds if it is disabled. Because we’ve found that the 60-second difference has no significant impact on test results, we are changing the recommendation to 60 seconds for both hpamode settings.


We hope these changes will improve the CloudXPRT setup and testing experience. We haven’t set the release date for the updated build yet, but when we do, we’ll announce it here in the blog. If you have any questions about CloudXPRT, or would like to report bugs or other issues, please feel free to contact us!

Justin

Check out the other XPRTs:

Forgot your password?