BenchmarkXPRT Blog banner

Category: Google Chrome

The CrXPRT 2 Community Preview is available!

We’re excited to announce that the CrXPRT 2 Community Preview (CP) is now available! BenchmarkXPRT Development Community members can access the preview using a direct link posted on the CrXPRT tab in the XPRT Members’ Area (login required), where they will also find the CrXPRT 2 CP user manual.

You can find more information about the key differences between CrXPRT 2015 and CrXPRT 2 in last week’s blog entry. During the preview period, we allow testers to publish CP test scores, but CrXPRT 2 overall performance test scores and battery life measurements are not comparable to CrXPRT 2015 scores.

If you have any questions about CrXPRT 2 or joining the community, please let us know!

Justin

The CrXPRT 2 Community Preview is almost here!

We appreciate everyone’s patience and feedback during the CrXPRT 2 development process. We’re excited to say that we’re now wrapping up some final details and expect to release the Community Preview (CP) within the next week.

Here is a summary of the key differences between CrXPRT 2015 and CrXPRT 2:

  • As we mentioned a few weeks ago, CrXPRT 2 has a completely new UI in line with the functional and aesthetic themes we used for the latest versions of WebXPRT, MobileXPRT, and HDXPRT, with a focus on intuitive navigation.
  • The CrXPRT 2 performance test includes six of the seven workloads in CrXPRT 2015. Newer versions of Chrome can’t run the Photo Collage workload without a workaround, so we removed it from CrXPRT 2.
  • We updated the images in the Photo Effects and Face Detection workloads to reflect more contemporary file resolutions and sizes.
  • The CrXPRT 2 battery life test requires a full rundown, so you’ll need charge your device to 100 percent before you can start the test, and the length of battery life tests will vary according to the battery life of the systems under test.
  • We no longer require testers to enter luminance and audio measurements in order to run a battery life test.
  • We added a second video playback segment to each battery life iteration.


We allow testers to publish CP test scores, but CrXPRT 2 overall performance test scores and battery life measurements are not comparable to CrXPRT 2015 scores.

Only BenchmarkXPRT Development Community members will be able to download the CrXPRT 2 CP. Because the Chrome team stopped providing search and browse functions for hosted and packaged Chrome apps in the Chrome Web Store, members will need a direct link to access the app. Once the app is available, we’ll post that link, along with the CrXPRT 2 CP user manual, on the CrXPRT tab in the XPRT Members’ Area (login required). We’ll also send a message to the community and post a notice here in the blog.

If you have any questions about CrXPRT 2 or joining the community, please let us know!

Justin

Using WebXPRT 3 to compare the performance of popular browsers

Microsoft recently released a new Chromium-based version of the Edge browser, and several tech press outlets have released reviews and results from head-to-head browser performance comparison tests. Because WebXPRT is a go-to benchmark for evaluating browser performance, PCMag, PCWorld, and VentureBeat, among others, used WebXPRT 3 scores as part of the evaluation criteria for their reviews.

We thought we would try a quick experiment of our own, so we grabbed a recent laptop from our Spotlight testbed: a Dell XPS 13 7930 running Windows 10 Home 1909 (18363.628) with an Intel Core i3-10110U processor and 4 GB of RAM. We tested on a clean system image after installing all current Windows updates, and after the update process completed, we turned off updates to prevent them from interfering with test runs. We ran WebXPRT 3 three times on six browsers: a new browser called Brave, Google Chrome, the legacy version of Microsoft Edge, the new version of Microsoft Edge, Mozilla Firefox, and Opera. The posted score for each browser is the median of the three test runs.

As you can see in the chart below, five of the browsers (legacy Edge, Brave, Opera, Chrome, and new Edge) produced scores that were nearly identical. Mozilla Firefox was the only browser that produced a significantly different score. The parity among Brave, Chrome, Opera, and the new Edge is not that surprising, considering they are all Chromium-based browsers. The rank order and relative scaling of these results is similar to the results published by the tech outlets mentioned above.

Do these results mean that Mozilla Firefox will provide you with a speedier web experience? Generally, a device with a higher WebXPRT score is probably going to feel faster to you during daily use than one with a lower score. For comparisons on the same system, however, the answer depends in part on the types of things you do on the web, how the extensions you’ve installed affect performance, how frequently the browsers issue updates and incorporate new web technologies, and how accurately the browsers’ default installation settings reflect how you would set up the same browsers for your daily workflow.

In addition, browser speed can increase or decrease significantly after an update, only to swing back in the other direction shortly thereafter. OS-specific optimizations can also affect performance, such as with Edge on Windows 10 and Chrome on Chrome OS. All of these variables are important to keep in mind when considering how browser performance comparison results translate to your everyday experience. In such a competitive market, and with so many variables to consider, we’re happy that WebXPRT can help consumers by providing reliable, objective results.

What are your thoughts on today’s competitive browser market? We’d love to hear from you.

Justin

Planning for the next CrXPRT

We’re currently planning the next version of CrXPRT, our benchmark that evaluates the performance and battery life of Chromebooks. If you’re unfamiliar with CrXPRT, you can find out more about how it works both here in the blog and at CrXPRT.com. If you’ve used CrXPRT, we’d love to hear any suggestions you may have. What do you like or dislike about CrXPRT? What features do you hope to see in a new version?

When we begin work on a new version of any benchmark, one of our first steps is to determine whether the workloads will provide value during the years ahead. As technology and user behavior evolve, we update test content to be more relevant. One example is when we replace photos with ones that use more contemporary file resolutions and sizes.

Sometimes the changing tech landscape prompts us to remove entire workloads and add new ones. The Photo Collage workload in CrXPRT uses Portable Native Client (PNaCl) technology, for which the Chrome team will soon end support. CrXPRT 2015 has a workaround for this issue, but the best course of action for the next version of CrXPRT will be to remove this workload altogether.

The battery life test will also change. Earlier this year, we started to see unusual battery life estimates and high variance when running tests at CrXPRT’s default battery life test length of 3.5 hours, so we’ve been recommending that users perform full rundowns instead. In the next CrXPRT, the battery life test will require full rundowns.

We’ll also be revamping the CrXPRT UI to improve the look of the benchmark and make it easier to use, as we’ve done with the other recent XPRT releases.

We really do want to hear your ideas, and any feedback you send has a chance to shape the future of the benchmark. Let us know what you think!

Justin

A CrXPRT fix for Chrome 76

After Chrome OS version 76 moved from Chrome’s Beta channel to the Stable channel last week, we became aware of an issue that occurs when CrXPRT’s Photo Collage workload runs on a Chrome 76 system. We found that the Photo Collage workload produces an error message—“This plugin is not supported on this device”—and the test run does not complete.

The error occurs because the Photo Collage workload uses Portable Native Client (PNaCl), and starting with version 76, the Chrome team changed the way the OS handles PNaCl tasks. Technically, Chrome still supports PNaCl, but the OS now disables the capability by default. Chrome’s current plan is to end support for PNaCl by the end of this year, focusing related development efforts on WebAssembly instead.

We’ll investigate the best path forward during this transition, but for now, testers can use the following workaround that allows CrXPRT to complete successfully. Simply navigate to chrome://flags on the test system, and find the Native Client flag, which is set to “Disabled” by default. Click the toggle switch to “Enabled” to allow native client capabilities, restart the system, and kick off a CrXPRT test in the normal manner.

We’ll update the CrXPRT web page and test documentation to include information about the workaround. In the long term, we’re interested in any suggestions you have for CrXPRT—whether they’re related to PNaCl or not. Please let us know your thoughts!

Justin

A new playing field for WebXPRT

WebXPRT is one of the go-to benchmarks for evaluating browser performance, so we’re always interested in browser development news. Recently, Microsoft created a development channel where anyone can download early versions of an all-new Microsoft Edge browser. Unlike previous versions of Edge, Microsoft constructed the new browser using the Chromium open-source project, the same foundation underlying the Google Chrome browser and Chrome OS.

One interesting aspect of the new Edge development strategy is the changes that Microsoft is making to more than 50 services that Chromium has included. If you use Chrome daily, you’ve likely become accustomed to certain built-in services such as ad block, spellcheck, translate, maps integration, and form fill, among many others. While each of these is useful, a large number of background services running simultaneously can slow browsing and sap battery life. In the new Edge, Microsoft is either reworking each service or removing it altogether, with the hope of winning users by providing a cleaner, faster, and more power-efficient experience. You can read more about Microsoft’s goals for the new project on the Microsoft Edge Insider site.

As we’ve discussed before, many factors contribute to the speed of a browsing experience and its WebXPRT score. It’s too early to know how the new Microsoft Edge will stack up against other browsers, but when the full version comes out of development, you can be sure that we’ll be publishing some comparison scores. I’ve installed the Dev Channel version of Edge on my personal machine and run WebXPRT 3. While I can’t publish the scores from this early version, I can tell you that the results were interesting. Have you run WebXPRT 3 on the new Microsoft Edge? How do you think it compares to competitors? We’d love to hear your thoughts.

Justin

Check out the other XPRTs:

Forgot your password?