BenchmarkXPRT Blog banner

Tag Archives: Chrome OS

Chrome OS support for CrXPRT apps ends in June 2022

Justin Greene
By Justin Greene
on March 24, 2022

Last March, we discussed the Chrome OS team’s original announcement that they would be phasing out support for Chrome Apps altogether in June 2021, and would shift their focus to Chrome extensions and Progressive Web Apps. The Chrome OS team eventually extended support for existing Chrome Apps through June 2022, but as of this week, we see no indication that they will further extend support for Chrome Apps published with general developer accounts. If the end-of-life schedule for Chrome Apps does not change in the next few months, both CrXPRT 2 and CrXPRT 2015 will stop working on new versions of Chrome OS at some point in June.

To maintain CrXPRT functionality past June, we would need to rebuild the app completely—either as a Progressive Web App or in some other form. For this reason, we want to reassess our approach to Chrome OS testing, and investigate which features and technologies to include in a new Chrome OS benchmark. Our current goal is to gather feedback and conduct exploratory research over the next few months, and begin developing an all-new Chrome OS benchmark for publication by the end of the year.

While we will discuss ideas for this new Chrome OS benchmark in future blog posts, we welcome ideas from CrXPRT users now. What features or workloads would you like the new benchmark to retain? Would you like us to remove any components from the existing benchmark? Does the battery life test in its current form suit your needs? If you have any thoughts about these questions or any other aspects of Chrome OS benchmarking, please let us know!

Justin

Posted in Battery life, battery life, benchmark, Benchmarking, BenchmarkXPRT, Browser-based benchmarks, Chrome OS, Chromebooks, Chromium, Collaborative benchmark development, CrXPRT, Google Chrome, Performance benchmarking, What makes a good benchmark? | Also tagged , , , , , , , ,

Updated system configuration recommendations for CrXPRT 2 battery life tests

Justin Greene
By Justin Greene
on February 3, 2022

Recently, we heard from a BenchmarkXPRT Development Community member who was testing Chromebooks in their lab. On a few of the Chromebooks, they saw sporadic CrXPRT 2 battery life test failures where CrXPRT 2 would successfully complete a battery life test and produce a result for the initial run, but then fail at the end of later runs.

After a considerable amount of troubleshooting, they determined that the issue seemed to be related to the way some systems automatically shut down before the battery is completely exhausted, and the way some systems will automatically boot up once the tester plugs in the power adapter for charging. This member found that when they added a few system configuration steps before battery life tests and made slight changes to their post-test routine, the systems that had previously experienced consistent failures would successfully complete battery life tests and produce results.

The added steps are quick and straightforward, and we decided to add them to the Configuring the test device and Running the tests sections of the CrXPRT 2 user manual. We hope this updated guidance will help to prevent future frustration for CrXPRT 2 testers.

If you have any questions or comments about the CrXPRT 2 battery life test, please feel free to contact us!

Justin

Posted in battery life, benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Chrome OS, Chromebooks, CrXPRT, Google Chrome | Also tagged , , , , , , ,

Why we don’t control screen brightness during CrXPRT 2 battery life tests

Justin Greene
By Justin Greene
on January 27, 2022

Recently, we had a discussion with a community member about why we no longer recommend specific screen brightness settings during CrXPRT 2 battery life tests. In the CrXPRT 2015 user manual, we recommended setting the test system’s screen brightness to 200 nits. Because the amount of power that a system directs to screen brightness can have a significant impact on battery life, we believed that pegging screen brightness to a common standard for all test systems would yield apple-to-apples comparisons.

After extensive experience with CrXPRT 2015 testing, we decided to not recommend a standard screen brightness with CrXPRT 2, for the following reasons:

  • A significant number of Chromebooks cannot produce a screen brightness of 200 nits. A few higher-end models can do so, but they are not representative of most Chromebooks. Some Chromebooks, especially those that many school districts and corporations purchase in bulk, cannot produce a brightness of even 100 nits.
  • Because of the point above, adjusting screen brightness would not represent real-life conditions for most Chromebooks, and the battery life results could mislead consumers who want to know the battery life they can expect with default out-of-box settings.
  • Most testers, and even some labs, do not have light meters, and the simple brightness percentages that the operating system reports produce different degrees of brightness on different systems. For testers without light meters, a standardized screen brightness recommendation could discourage them from running the test.
  • The brightness controls for some low-end Chromebooks lack the fine-tuning capability that is necessary to standardize brightness between systems. In those cases, an increase or decrease of one notch can swing brightness by 20 to 30 nits in either direction. This could also discourage testing by leading people to believe that they lack the capability to correctly run the test.

In situations where testers want to compare battery life using standardized screen brightness, we recommend using light meters to set the brightness levels as closely as possible. If the brightness levels between systems vary by more than few nits, and if the levels vary significantly from out-of-box settings, the publication of any resulting battery life results should include a full disclosure and explanation of test conditions.

For the majority of testers without light meters, running the CrXPRT 2 battery life test with default screen brightness settings on each system provides a reliable and accurate estimate of the type of real-world, out-of-box battery life consumers can expect.

If you have any questions or comments about the CrXPRT 2 battery life test, please feel free to contact us!

Justin

Posted in Battery life, battery life, benchmark, Benchmark metrics, Chrome OS, Chromebooks, CrXPRT, What makes a good benchmark? | Also tagged , , , , ,

The CrXPRT 2 battery life test is back!

Justin Greene
By Justin Greene
on October 21, 2021

Last month, we discussed a potential fix for the error that was preventing CrXPRT 2 testers from successfully completing battery life tests on systems running Chrome v89.x and later. Since then, we’ve been testing an updated, unpublished version of the app package across several Chromebook models to ensure that the new build is stable and produces consistent results. We’re happy to report that our testing was successful, and we’ve published the new CrXPRT build (v1.2.0.0) in the Chrome Web Store and it is live as of 12:45 PM EDT today.

Note that it might take some time for the update to appear on your Chromebook and, once it does, you might have to manually approve the update notice.

Neither the tests nor the method of calculating the overall score and battery-life score in this new build have changed, so results are comparable with previous CrXPRT 2 results.

We appreciate everyone’s patience while we found a solution to the error. If you have any questions or comments about the CrXPRT 2 battery life test, please feel free to contact us!

Justin

Posted in Battery life, battery life, benchmark, Benchmark metrics, Chrome OS, Chromebooks, CrXPRT, Google Chrome | Also tagged , , , , ,

A potential fix for the CrXPRT 2 battery life test error

Justin Greene
By Justin Greene
on September 30, 2021

For the past few months, we’ve been recommending that CrXPRT 2 testers not use the battery life test until we find a solution to a recurring error on Chrome v89.x and later. The error prevents the test from completing and producing a battery life estimate. Sometimes, the CrXPRT battery life test stops running after only a few workload iterations, while at other times, it almost reaches completion before producing the error.

We are cautiously optimistic that we’ve identified both the problem and a potential fix. We believe the problem stems from fluctuations in the time it takes the benchmark to communicate with Chrome to collect and store battery life information. While we haven’t identified the root cause of the fluctuations, adjusting the CrXPRT code to make it less sensitive to the fluctuations appears to be an effective fix. We have incorporated those adjustments into an updated, unpublished version of the app package, and we can now complete CrXPRT 2 battery life tests on Chrome v89.x and later with no failures.

We are calling this a potential fix because we’re still testing across several different Chromebook models to ensure consistency. In some testing, the variance in estimated battery life results has been a little higher than we like, so we’re taking time to determine whether that variance is present across all systems or on only specific hardware.

We’d like to apologize once again for the inconvenience that this error is causing CrXPRT 2 testers. As soon as we better understand the viability of the current fix as a long-term update, we’ll let you know!

Justin

Posted in battery life, benchmark, BenchmarkXPRT, Browser-based benchmarks, Chrome OS, Chromebooks, CrXPRT, results | Also tagged , , , ,

Persistent CrXPRT 2 battery life test error on Chrome v89 and later

Justin Greene
By Justin Greene
on June 17, 2021

A few weeks ago, we discussed an error that we’d recently started encountering during the CrXPRT 2 battery life test on systems running Chrome OS v89.x and later. The error prevents the test from completing and producing a battery life estimate. CrXPRT stops running its normal workload cycle and produces a “Test Error” page. The timing of the error can vary from run to run. Sometimes, CrXPRT stops running after only a few workload iterations, while other times, the battery life test almost reaches completion before producing the error.

We have seen the error on across multiple brands of Chromebooks running Chrome OS v89.x and later. To our knowledge, Chromebooks running Chrome OS v88.x and earlier versions complete the battery life test without issues. We are unaware of any problems with the CrXPRT 2 performance test.

We’re continuing to investigate this problem. Unfortunately, we have not yet identified the root cause. Without a solution, we are recommending that for now, testers not use the CrXPRT 2 battery life test. We will post this recommendation on CrXPRT.com.

We apologize for the inconvenience that this error is causing CrXPRT 2 testers. As soon as we identify a possible solution, we will share that information here in the blog. If you have any insight into recent Chrome OS changes or flag settings that could be causing this problem, please let us know!

Justin

Posted in battery life, benchmark, Chrome OS, Chromebooks, Chromium, CrXPRT, Google Chrome | Also tagged , , , ,

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2024   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x