Category: Chromium

An update on Chrome OS support for CrXPRT

on July 14, 2022

In March, we discussed the Chrome OS team’s plan to end support for Chrome apps in June and instead focus their efforts on Chrome extensions and Progressive Web Apps. After receiving feedback on their published timeline, the Chrome OS team decided to extend Chrome app support for Enterprise and Education account customers through January 2025. Because we publish our Chrome app (CrXPRT) through a private BenchmarkXPRT developer account, and because we have not seen any further updates to the support timeline, we don’t assume that the support extension will apply to CrXPRT.

Since June has come and gone, and the support extension probably does not apply to our account, we do not expect to be able to publish any future fixes or updates for CrXPRT. As of now, and up through Chrome 105, the CrXPRT 2 performance and battery life tests are still working without a hitch. We will continue to run the benchmark on a regular basis to monitor functionality, and we will disclose any future issues here in the blog and on CrXPRT.com. We hope the app will continue to run both performance and battery life tests well into the future. However, given the frequency of Chrome updates, it’s difficult for us to predict how long the benchmark will remain viable.

As we mentioned back in March, we hope to begin development of an all-new Chrome OS XPRT benchmark by the end of this year. We’ll discuss that prospect in more detail in future blog posts, but if you have ideas about the types of features or workloads you’d like to see in a new Chrome OS benchmark, please let us know!

Justin

Posted in Battery life, battery life, benchmark, BenchmarkXPRT, Chrome OS, Chromebooks, Chromium, CrXPRT, Google Chrome | Tagged battery life, benchmark, BenchmarkXPRT, Chrome, Chrome OS, Chrome Web Store, Chromebooks, extensions, Progressive Web Apps |

Using WebXPRT 4 to compare the performance of popular browsers

By Justin Greene

on April 14, 2022

From time to time, we like to run a series of in-house WebXPRT comparison tests to see if recent updates have changed the performance rankings of popular web browsers. We published our most recent comparison last October, when we used WebXPRT 3 to compare Windows 10 and Windows 11 browser performance on the same system. Now that WebXPRT 4 is live, it’s time to update our comparison series with the newest member of the XPRT family.

For this round of tests, we used a Dell XPS 13 7930, which features an Intel Core i3-10110U processor and 4 GB of RAM, running Windows 11 Home updated to version 21H2 (22000.593). We installed all current Windows updates and tested on a clean system image. After the update process completed, we turned off updates to prevent them from interfering with test runs. We ran WebXPRT 4 three times each across five browsers: Brave, Google Chrome, Microsoft Edge, Mozilla Firefox, and Opera. The posted score for each browser is the median of the three test runs.

In our previous round of tests with WebXPRT 3, Google Chrome narrowly beat out Firefox in Windows 10 and Windows 11 testing, but the scores among three of the Chromium-based browsers (Chrome, Edge, and Opera) were close enough that most users performing common daily tasks would be unlikely to notice a difference. Brave performance lagged by about 7 percent, a difference that may be noticeable to most users. This time, when testing updated versions of the browsers with WebXPRT 4 on Windows 11, the rankings changed. Edge was the clear winner, with a 2.2 percent performance advantage over Chrome. Firefox came in last, about 3 percent slower than Opera, which was in the middle of the pack. Performance from Brave improved to the point that it was no longer lagging the other Chromium-based browsers.

Do these results mean that Microsoft Edge will always provide you with a speedier web experience? A device with a higher WebXPRT score will probably feel faster during daily use than one with a lower score. For comparisons on the same system, however, the answer depends in part on the types of things you do on the web, how the extensions you’ve installed affect performance, how frequently the browsers issue updates and incorporate new web technologies, and how accurately each browser’s default installation settings reflect how you would set up that browser for your daily workflow.

In addition, browser speed can increase or decrease significantly after an update, only to swing back in the other direction shortly thereafter. OS-specific optimizations can also affect performance, such as with Edge on Windows 11 and Chrome on Chrome OS. All these variables are important to keep in mind when considering how WebXPRT results translate to your everyday experience.

Do you have insights you’d like to share from using WebXPRT to compare browser performance? Let us know!

Justin

Posted in benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, Brave, Browser-based benchmarks, Chromium, Cross-platform benchmarks, Firefox, Google Chrome, Microsoft Edge, Mozilla, Opera, Performance benchmarking, results, WebXPRT, WebXPRT 3, webxprt 4 | Tagged Brave, browser, browser benchmark, browser performance, Chrome, Firefox, Microsoft Edge, Opera, WebXPRT, WebXPRT 3, WebXPRT 4 |

Chrome OS support for CrXPRT apps ends in June 2022

By Justin Greene

on March 24, 2022

Last March, we discussed the Chrome OS team’s original announcement that they would be phasing out support for Chrome Apps altogether in June 2021, and would shift their focus to Chrome extensions and Progressive Web Apps. The Chrome OS team eventually extended support for existing Chrome Apps through June 2022, but as of this week, we see no indication that they will further extend support for Chrome Apps published with general developer accounts. If the end-of-life schedule for Chrome Apps does not change in the next few months, both CrXPRT 2 and CrXPRT 2015 will stop working on new versions of Chrome OS at some point in June.

To maintain CrXPRT functionality past June, we would need to rebuild the app completely—either as a Progressive Web App or in some other form. For this reason, we want to reassess our approach to Chrome OS testing, and investigate which features and technologies to include in a new Chrome OS benchmark. Our current goal is to gather feedback and conduct exploratory research over the next few months, and begin developing an all-new Chrome OS benchmark for publication by the end of the year.

While we will discuss ideas for this new Chrome OS benchmark in future blog posts, we welcome ideas from CrXPRT users now. What features or workloads would you like the new benchmark to retain? Would you like us to remove any components from the existing benchmark? Does the battery life test in its current form suit your needs? If you have any thoughts about these questions or any other aspects of Chrome OS benchmarking, please let us know!

Justin

Posted in Battery life, battery life, benchmark, Benchmarking, BenchmarkXPRT, Browser-based benchmarks, Chrome OS, Chromebooks, Chromium, Collaborative benchmark development, CrXPRT, Google Chrome, Performance benchmarking, What makes a good benchmark? | Tagged battery life, benchmark, BenchmarkXPRT, Chrome, Chrome OS, Chrome Web Store, Chromebook, CrXPRT, CrXPRT 2, CrXPRT 2015 |

Persistent CrXPRT 2 battery life test error on Chrome v89 and later

By Justin Greene

on June 17, 2021

A few weeks ago, we discussed an error that we’d recently started encountering during the CrXPRT 2 battery life test on systems running Chrome OS v89.x and later. The error prevents the test from completing and producing a battery life estimate. CrXPRT stops running its normal workload cycle and produces a “Test Error” page. The timing of the error can vary from run to run. Sometimes, CrXPRT stops running after only a few workload iterations, while other times, the battery life test almost reaches completion before producing the error.

We have seen the error on across multiple brands of Chromebooks running Chrome OS v89.x and later. To our knowledge, Chromebooks running Chrome OS v88.x and earlier versions complete the battery life test without issues. We are unaware of any problems with the CrXPRT 2 performance test.

We’re continuing to investigate this problem. Unfortunately, we have not yet identified the root cause. Without a solution, we are recommending that for now, testers not use the CrXPRT 2 battery life test. We will post this recommendation on CrXPRT.com.

We apologize for the inconvenience that this error is causing CrXPRT 2 testers. As soon as we identify a possible solution, we will share that information here in the blog. If you have any insight into recent Chrome OS changes or flag settings that could be causing this problem, please let us know!

Justin

Posted in battery life, benchmark, Chrome OS, Chromebooks, Chromium, CrXPRT, Google Chrome | Tagged battery life, Chrome, Chrome OS, Chromebooks, CrXPRT, CrXPRT 2 |

CrXPRT support through 2022

By Justin Greene

on March 18, 2021

CrXPRT testers may remember that back around the time that we began the CrXPRT 2 development process, the Chrome team announced that they were phasing out support for Portable Native Client (PNaCL) in favor of WebAssembly (WASM). As a first step, they changed the Chrome OS setting that enabled PNaCL by default. At the time, this caused problems with the Photo Collage workload in CrXPRT 2015, and even though we identified a workaround, details in the Chrome team’s announcement led us to conclude that the workaround might stop working in June 2021. Because of this change, we decided that the best decision would be to remove the workload from CrXPRT 2, and keep existing CrXPRT 2015 testers informed of any changes with the workaround.

In 2020, the Chrome team also announced that they would be phasing out support for Chrome Apps altogether starting in June 2021, and would shift their focus to Chrome extensions. This change would have required us to reassess the viability of CrXPRT in anything like its current form.

We’re happy to report that the Chrome team has extended support for PNaCL and existing Chrome Apps through June 2022. Barring further changes, this means that CrXPRT 2015 (with the workaround) and CrXPRT 2 should continue to serve as reliable Chrome OS evaluation tools for some time.

If you have any questions about CrXPRT 2, please let us know!

Justin

Posted in benchmark, Benchmarking, BenchmarkXPRT, Browser-based benchmarks, Chrome OS, Chromium, CrXPRT, Google Chrome, Performance benchmarking, PNaCL, WebAssembly | Tagged Chrome, Chrome OS, Chromebooks, Chromium, CrXPRT, CrXPRT 2, PNaCl, WebAssembly |

Using WebXPRT 3 to compare the performance of popular browsers (Round 2)

By Justin Greene

on November 5, 2020

It’s been nine months since we’ve published a WebXPRT 3 browser performance comparison, so we decided to put the newest versions of popular browsers through the paces to see if the performance rankings have changed since our last round of tests.

We used the same laptop as last time: a Dell XPS 13 7930 with an Intel Core i3-10110U processor and 4 GB of RAM running Windows 10 Home, updated to version 1909 (18363.1139). We installed all current Windows updates and tested on a clean system image. After the update process completed, we turned off updates to prevent them from interfering with test runs. We ran WebXPRT 3 three times on five browsers: Brave, Google Chrome, Microsoft Edge, Mozilla Firefox, and Opera. The posted score for each browser is the median of the three test runs.

In our last round of tests, the four Chromium-based browsers (Brave, Chrome, Edge, and Opera) produced scores that were nearly identical. Only Mozilla Firefox produced a significantly different (and better) score. The parity of the Chromium-based browsers was not surprising, considering they have the same underlying foundation.

In this round of testing, the Chromium-based browsers again produced very close scores, although Brave’s performance lagged by about 4 percent. Firefox again separated itself from the pack with a higher score. With the exception of Chrome, which produced an identical score as last time, every browser’s score was slightly slower than before. There are many possible reasons for this, including increased overhead in the browsers or changes in Windows, and the respective slowdowns for each browser will probably be unnoticeable to most users during everyday tasks.

Do these results mean that Mozilla Firefox will provide you with a speedier web experience? As we noted in the last comparison, a device with a higher WebXPRT score will probably feel faster during daily use than one with a lower score. For comparisons on the same system, however, the answer depends in part on the types of things you do on the web, how the extensions you’ve installed affect performance, how frequently the browsers issue updates and incorporate new web technologies, and how accurately each browsers’ default installation settings reflect how you would set up that browser for your daily workflow.

In addition, browser speed can increase or decrease significantly after an update, only to swing back in the other direction shortly thereafter. OS-specific optimizations can also affect performance, such as with Edge on Windows 10 and Chrome on Chrome OS. All of these variables are important to keep in mind when considering how browser performance comparison results translate to your everyday experience.

What are your thoughts on browser performance? Let us know!

Justin

Posted in benchmark, Benchmarking, Brave, Chromium, Cross-platform benchmarks, Firefox, Google Chrome, Microsoft Edge, Mozilla, Opera, Performance benchmarking, WebXPRT, WebXPRT 3, Windows 10 | Tagged Brave, Chrome, Firefox, Google Chrome, Microsoft Edge, Mozilla, Opera, WebXPRT, WebXPRT 3 |

Category: Chromium

An update on Chrome OS support for CrXPRT

Using WebXPRT 4 to compare the performance of popular browsers

Chrome OS support for CrXPRT apps ends in June 2022

Persistent CrXPRT 2 battery life test error on Chrome v89 and later

CrXPRT support through 2022

Using WebXPRT 3 to compare the performance of popular browsers (Round 2)

Check out the other XPRTs: