Last month, we discussed a potential fix for the error that was preventing CrXPRT 2 testers from successfully completing battery life tests on systems running Chrome v89.x and later. Since then, we’ve been testing an updated, unpublished version of the app package across several Chromebook models to ensure that the new build is stable and produces consistent results. We’re happy to report that our testing was successful, and we’ve published the new CrXPRT build (v1.2.0.0) in the Chrome Web Store and it is live as of 12:45 PM EDT today.
Note
that it might take some time for the update to appear on your Chromebook and,
once it does, you might have to manually approve the update notice.
Neither
the tests nor the method of calculating the overall score and battery-life
score in this new build have changed, so results are comparable with previous
CrXPRT 2 results.
We appreciate everyone’s patience while we found a solution to the error. If you have any questions or comments about the CrXPRT 2 battery life test, please feel free to contact us!
For
the past few months, we’ve been recommending that CrXPRT 2
testers not use the battery life test until we find a solution to a recurring
error on Chrome v89.x and later. The error prevents the test from completing
and producing a battery life estimate. Sometimes, the CrXPRT battery life test stops
running after only a few workload iterations, while at other times, it almost
reaches completion before producing the error.
We are cautiously optimistic that we’ve identified both the problem and a potential fix. We believe the problem stems from fluctuations in the time it takes the benchmark to communicate with Chrome to collect and store battery life information. While we haven’t identified the root cause of the fluctuations, adjusting the CrXPRT code to make it less sensitive to the fluctuations appears to be an effective fix. We have incorporated those adjustments into an updated, unpublished version of the app package, and we can now complete CrXPRT 2 battery life tests on Chrome v89.x and later with no failures.
We
are calling this a potential fix because we’re still testing across several
different Chromebook models to ensure consistency. In some testing, the
variance in estimated battery life results has been a little higher than we
like, so we’re taking time to determine whether that variance is present across
all systems or on only specific hardware.
We’d like to apologize once again for the inconvenience that this error is causing CrXPRT 2 testers. As soon as we better understand the viability of the current fix as a long-term update, we’ll let you know!
A few weeks ago, we discussed an error that we’d recently started encountering during the CrXPRT 2 battery life test on systems running Chrome OS v89.x and later.
The error prevents the test from completing and producing a battery life
estimate. CrXPRT stops running its normal workload cycle and produces a “Test
Error” page. The timing of the error can vary from run to run. Sometimes,
CrXPRT stops running after only a few workload iterations, while other times,
the battery life test almost reaches completion before producing the error.
We have seen the error on across multiple brands of Chromebooks running
Chrome OS v89.x and later. To our knowledge, Chromebooks running Chrome OS v88.x
and earlier versions complete the battery life test without issues. We are unaware
of any problems with the CrXPRT 2 performance test.
We’re continuing to investigate this problem. Unfortunately, we have not yet identified the root cause. Without a solution, we are recommending that for now, testers not use the CrXPRT 2 battery life test. We will post this recommendation on CrXPRT.com.
We apologize for the inconvenience that this error is causing CrXPRT 2 testers. As soon as we identify a possible solution, we will share that information here in the blog. If you have any insight into recent Chrome OS changes or flag settings that could be causing this problem, please let us know!
In recent lab tests, we’ve encountered an error during the CrXPRT 2 battery life test that prevents the test from completing and producing a battery life estimate. As the screenshot below shows, when the error occurs, CrXPRT stops running its normal workload cycle and produces a “Test Error” page. We have seen this behavior on systems running Chrome OS v89.x and v90.x, across multiple vendor platforms. In our testing, Chromebooks running Chrome OS v88.x and earlier versions continue to complete the battery life test without any issues.
The error occurs consistently on every Chromebook running v89.x or
v90.x that we’ve tested so far. However, the timing of the error varies from
run to run on the same system. Sometimes, CrXPRT stops running after only a few
workload iterations, while at other times, the battery life test runs almost to
completion before producing the error.
We’re actively investigating this problem, but have not yet
identified the root cause. We apologize for the inconvenience that this error
may be causing CrXPRT 2 testers. As soon as we identify the root cause of the
problem and have ideas about possible solutions, we will share that information
here in the blog. If you have any insight into recent Chrome OS changes or flag
settings that could be causing this problem, please let us know!
CrXPRT testers may
remember that back around the time that we began the CrXPRT 2 development process, the Chrome team announced that they were
phasing out support for Portable Native Client (PNaCL) in favor of WebAssembly (WASM). As a first step,
they changed the Chrome OS setting that enabled PNaCL by default. At the time,
this caused problems with the Photo Collage workload in CrXPRT 2015, and even
though we identified a workaround, details in the Chrome team’s announcement led us to conclude
that the workaround might stop working in June 2021. Because of this change, we
decided that the best decision would be to remove the workload from CrXPRT
2, and keep existing CrXPRT 2015 testers informed of any changes with the
workaround.
In 2020, the Chrome
team also announced that they would be phasing out support for Chrome Apps
altogether starting in June 2021, and would shift their focus to Chrome
extensions. This change would have required us to reassess the viability of
CrXPRT in anything like its current form.
We’re happy to report that
the Chrome team has extended support for PNaCL and existing Chrome Apps through
June 2022. Barring further changes, this means that CrXPRT
2015 (with the workaround) and CrXPRT 2 should continue to serve as reliable
Chrome OS evaluation tools for some time.
If you have any questions about CrXPRT 2, please let us know!
It’s been nine months
since we’ve published a WebXPRT 3 browser performance comparison, so we decided
to put the newest versions of popular browsers through the paces to see if the
performance rankings have changed since our last round of tests.
We used the same
laptop as last time: a Dell XPS 13 7930 with an Intel Core i3-10110U processor and 4 GB of RAM
running Windows 10 Home, updated to version 1909 (18363.1139). We installed all
current Windows updates and tested on a clean system image. After the update
process completed, we turned off updates to prevent them from interfering with
test runs. We ran WebXPRT 3 three times on five browsers: Brave, Google Chrome,
Microsoft Edge, Mozilla Firefox, and Opera. The posted score for each browser
is the median of the three test runs.
In our last round of tests, the four Chromium-based browsers (Brave, Chrome, Edge, and Opera)
produced scores that were nearly identical. Only Mozilla Firefox produced a
significantly different (and better) score. The parity of the Chromium-based
browsers was not surprising, considering they have the same underlying foundation.
In this round of testing, the Chromium-based browsers again produced very close scores, although Brave’s performance lagged by about 4 percent. Firefox again separated itself from the pack with a higher score. With the exception of Chrome, which produced an identical score as last time, every browser’s score was slightly slower than before. There are many possible reasons for this, including increased overhead in the browsers or changes in Windows, and the respective slowdowns for each browser will probably be unnoticeable to most users during everyday tasks.
Do these results mean that Mozilla Firefox will provide you with a speedier web experience? As we noted in the last comparison, a device with a higher WebXPRT score will probably feel faster during daily use than one with a lower score. For comparisons on the same system, however, the answer depends in part on the types of things you do on the web, how the extensions you’ve installed affect performance, how frequently the browsers issue updates and incorporate new web technologies, and how accurately each browsers’ default installation settings reflect how you would set up that browser for your daily workflow.
In
addition, browser speed can increase or decrease significantly after an update,
only to swing back in the other direction shortly thereafter. OS-specific
optimizations can also affect performance, such as with Edge on Windows 10 and
Chrome on Chrome OS. All of these variables are important to keep in mind when
considering how browser performance comparison results translate to your
everyday experience.
What are your thoughts on browser performance? Let us know!
Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.
