BenchmarkXPRT Blog banner

Tag Archives: battery life

Feedback from the WebXPRT 4 tech press survey

Justin Greene
By Justin Greene
on June 10, 2021

In early May, we sent a survey to members of the tech press who regularly use WebXPRT in articles and reviews. We asked for their thoughts on several aspects of WebXPRT, as well as what they’d like to see in the upcoming fourth version of the benchmark. We also published the survey questions here in the blog, and invited experienced WebXPRT testers to send their feedback as well. We received some good responses to the survey, and for the benefit of our readers, we’ve summarized some of the key comments and suggestions below.

  • One respondent stated that WebXPRT is demanding enough to test performance, but if we want to simulate modern web usage, we should find the most up-to-date studies on common browser tasks and web technologies. This suggestion lines up with our intention to study the feasibility of adding a WebAssembly workload. 
  • One respondent liked that fact that unlike many other browser benchmarks, WebXPRT tests more than just JavaScript calculation speed.
  • One respondent suggested that we include a link to a WebXPRT white paper within the UI, or at least a guide describing what happens during each workload.
  • One respondent stated that they would like for WebXPRT to automatically produce a good result file on the local test system.
  • One respondent said that WebXPRT has a relatively long runtime for a browser benchmark, and they would prefer that the runtime not increase in WebXPRT 4.
  • We had no direct calls for a battery life test, because many testers already have scripts and/or methodologies in place for battery testing, but one tester suggested adding the ability to loop the test so users can measure performance over varying lengths of time.
  • There were no requests to bring back any aspects of WebXPRT 2015 that we removed in WebXPRT 3.
  • There were no reports of significant connection issues when testing with WebXPRT.

We greatly appreciate the members of the tech press that responded to the survey. We’re still in the planning stages of WebXPRT 4, so there’s still time for anyone to send comments or ideas to benchmarkxprtsupport@principledtechnologies.com. We look forward to hearing from you!

Justin

Posted in battery life, benchmark, Benchmarking, Browser-based benchmarks, Collaborative benchmark development, Performance benchmarking, Web-based testing, WebAssembly, WebXPRT, WebXPRT 3, What makes a good benchmark?, White papers | Also tagged , , , , , ,

CrXPRT 2 battery life error on Chrome 89 and 90

Justin Greene
By Justin Greene
on May 20, 2021

In recent lab tests, we’ve encountered an error during the CrXPRT 2 battery life test that prevents the test from completing and producing a battery life estimate. As the screenshot below shows, when the error occurs, CrXPRT stops running its normal workload cycle and produces a “Test Error” page. We have seen this behavior on systems running Chrome OS v89.x and v90.x, across multiple vendor platforms. In our testing, Chromebooks running  Chrome OS v88.x and earlier versions continue to complete the battery life test without any issues.

The error occurs consistently on every Chromebook running v89.x or v90.x that we’ve tested so far. However, the timing of the error varies from run to run on the same system. Sometimes, CrXPRT stops running after only a few workload iterations, while at other times, the battery life test runs almost to completion before producing the error.

We’re actively investigating this problem, but have not yet identified the root cause. We apologize for the inconvenience that this error may be causing CrXPRT 2 testers. As soon as we identify the root cause of the problem and have ideas about possible solutions, we will share that information here in the blog. If you have any insight into recent Chrome OS changes or flag settings that could be causing this problem, please let us know!

Justin

Posted in battery life, benchmark, Chrome OS, Chromebooks, CrXPRT, Google Chrome | Also tagged , , , , ,

Considering a battery life test for WebXPRT 4

Justin Greene
By Justin Greene
on April 8, 2021

A few weeks ago, we discussed the beginnings of a WebXPRT 4 development plan, and asked for reader feedback about potential workload changes. So far, the two most common feedback topics have been the possible addition of a WebAssembly workload, and the feasibility of including a browser-based battery life test. Today, we discuss what a WebXPRT 4 battery life test would look like, and some of the challenges we’d have to overcome to make it a reality.

Battery life tests fall into two primary categories: simple rundown tests and performance-weighted tests. Simple rundown tests measure battery life during extreme idle periods and loops of movie playbacks, etc., but do not reflect the wide-ranging mix of activities that characterize a typical day for most users. While they can be useful for performing very specific apple-to-apples comparisons, these tests have limited value when it comes to giving consumers a realistic estimation of the battery life they would experience during everyday use.

In contrast, performance-weighted battery life tests, such as the one in CrXPRT 2, attempt to reflect real-world usage. The CrXPRT battery life test simulates common daily usage patterns for Chromebooks by including all the productivity workloads from the performance test, plus video playback, audio playback, and gaming scenarios. It also includes periods of wait/idle time. We believe this mixture of diverse activity and idle time better represents typical real-life behavior patterns. This makes the resulting estimated battery life much more helpful for consumers who are trying to match a device’s capabilities with their real-world needs.

From a technical standpoint, WebXPRT’s cross-platform nature presents us with several challenges that we did not face while developing the CrXPRT battery life test for Chrome OS. While the WebXPRT performance tests run in almost any browser, cross-browser differences in battery life reporting may restrict the battery life test to a single browser. For instance, Mozilla has deprecated the battery status API for Firefox, and we’re not yet sure if there’s another approach that would work. If a WebXPRT 4 battery life test supported only a single browser, such as Chrome or Safari, would you still be interested in using it? Please let us know.

A browser-based battery life workflow also presents other challenges that we do not face in native client applications such as CrXPRT:

  • A browser-based battery life test would require the user to check the starting and ending battery capacities, with no way for the app to independently verify data accuracy.
  • The battery life test could require more babysitting in the event of network issues. We can catch network failures and try to handle them by reporting periods of network disconnection, but those interruptions could influence the battery life duration.
  • The factors above could make it difficult to achieve repeatability. One way to address that problem would be to run the test in a standardized lab environment with a steady internet connection, but a long list of standardized environmental requirements would make the battery life test less attractive and less accessible to many testers.

Our intention with today’s blog is not to make a WebXPRT 4 battery life test seem like an impossibility. Rather, we want to share our perspective on what the test might look like, and describe some of the challenges and considerations in play. If you have thoughts about battery life testing, or experience with battery life APIs in one or more of the major browsers, we’d love to hear from you!

Justin

Posted in battery life, benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, Browser-based benchmarks, Chrome OS, Collaborative benchmark development, Cross-platform benchmarks, CrXPRT, Mozilla, Web-based testing, WebAssembly, WebXPRT, webxprt 4 | Also tagged , , , , , , , , , ,

Moving forward with WebXPRT 4

Justin Greene
By Justin Greene
on March 4, 2021

In the coming months, we’ll be moving forward with the first stages of the WebXPRT 4 development process. It’s been a while since we last asked readers to send their thoughts about web technologies and workloads that may be a good fit for WebXPRT 4, but we’re still very much open to ideas. If you missed our previous posts about possible changes for WebXPRT 4, we recap the most prominent ideas below. We also request specific feedback regarding a potential battery life component.

  • Community members have asked about a WebXPRT 4 battery life test. Any such test would likely be very similar to the performance-weighted battery life test in CrXPRT 2 (as opposed to a simple rundown test). While WebXPRT runs in almost any browser, cross-browser compatibility issues could cause a WebXPRT battery life test to run in only one browser. If this turned out to be the case, would you still be interested in using the battery life test? Please let us know.
  • One of the most promising ideas is the potential addition of one or more WebAssembly (WASM) workloads. WASM is a low-level, binary instruction format that works across all modern browsers. It offers web developers a great deal of flexibility and provides the speed and efficiency necessary for running complex client applications in the browser. WASM enables a variety of workload scenario options, including gaming, video editing, VR, virtual machines, image recognition, and interactive educational content.
  • We are also considering adding a web-based machine learning workload with TensorFlow for JavaScript (TensorFlow.js). TensorFlow.js offers pre-trained models for a wide range of tasks including image classification, object detection, sentence encoding, and natural language processing. We could also use this technology to enhance one of WebXPRT’s existing AI-themed workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.
  • Other ideas include using a WebGL-based workload to target GPUs, and simulating common web applications.

We’ll start work on WebXPRT 4 soon, but there’s still time to send your comments and ideas, so please do so as quickly as possible!

Justin

Posted in battery life, benchmark, BenchmarkXPRT, Browser-based benchmarks, Collaborative benchmark development, Future of performance evaluation, Performance benchmarking, TensorFlow, WebAssembly, WebGL, WebXPRT, WebXPRT 3 | Also tagged , , , , , , , ,

Following up on CrXPRT 2 battery life errors

Justin Greene
By Justin Greene
on May 21, 2020

A few weeks ago, we discussed error messages that a tester received when starting up CrXPRT 2 after a battery life test. CrXPRT 2 battery life tests require a full battery rundown, after which the tester plugs in the Chromebook, turns it on, opens the CrXPRT 2 app, and sees the test results. In the reported cases, the tester opened the app after a battery life test that seemed successful, but saw “N/A” or “test error” messages instead of the results they expected.

During discussions about the end-of-test system environment, we realized that some testers might be unclear about how to tell that the battery has fully run down. During the system idle portion of CrXPRT 2 battery life test iterations, the Chromebook screen turns black and a small cursor appears somewhere on the screen to let testers know the test is still in progress. We believe that some testers, seeing the black screen but not the cursor, believe the system has shut down. Restarting CrXPRT 2 before the battery life test is complete could explain some of the “N/A” or “test error” messages users have reported.

If you see a black screen without a cursor, you can check to see whether the test is complete by looking for the small system power indicator light on the side or top of most Chromebooks. These are usually red, orange, or green, but if a light of any color is lit, the test is still underway. When the light goes out, the test has ended. You can plug the system in and power it on to see results.

Note that some Chromebooks provide low-battery warnings onscreen. During CrXPRT 2 battery life runs, testers should ignore these.

We hope this clears up any confusion about how to know when a CrXPRT 2 battery life test has ended. If you’ve received repeated “N/A” or “test error” messages during your CrXPRT 2 testing and the information above does not help, please let us know!

Justin

Posted in Battery life, battery life, benchmark, Benchmarking, Chrome OS, Chromebooks, CrXPRT, Google Chrome | Also tagged , , ,

Results details and unexpected behavior with the CrXPRT 2 battery life test

Justin Greene
By Justin Greene
on April 30, 2020

It’s been two weeks since the CrXPRT 2 public release, and we’re happy to see widespread interest in the test right out of the gate!

This week, we received a couple of questions about its battery life test from Melissa Riofrio at PCWorld. First, she asked for clarification about the relationship between the rundown time and the 30-minute increments that appear in the iteration details table for each battery life run. Second, she asked what could be causing her to get “N/A” and “test error” battery life results at the end of what appeared to be successful tests. Both topics may be of interest to other CrXPRT 2 testers, so we’ve decided to address them here in the blog and invite our readers to provide any relevant feedback.

Rundown time vs. elapsed time

When you’re viewing previous CrXPRT 2 test results and click the Details link for a specific battery life test run, a window displaying additional test information appears (the screenshot below shows an example). The window first provides performance details, then presents a table with several data points for each iteration.

The data point in the far-right column, elapsed time, can be slightly confusing. Each test iteration runs for 30 minutes, and this column provides a cumulative total of these 30-minute increments. In some instances, these totals accurately reflect the actual time elapsed from the time that testing begins. However, if the test system shuts down for some reason before running the entire iteration, this table will still show the entire 30 minutes allotted for the that iteration. In these cases, the cumulative elapsed time value in the far-right column will not match the rundown time that the test reports for the system’s battery life. For that reason, testers should always consider rundown time as the definitive value for battery life.


 “N/A” and “test error” battery life results after apparently successful tests

We’re actively investigating this issue at present. We’ve tested a wide range of Chromebooks, both old and new, on several versions of Chrome OS, including the latest versions, and have been unable to reproduce the problem. Have you witnessed this behavior at the end of a CrXPRT 2 battery life test? If so, we’d love to get more information from you about the system under test and your testing procedures, so please contact us.

We’re grateful to Melissa for raising these questions, and we appreciate everyone’s feedback on CrXPRT 2. Hopefully, we’ll soon be able to determine the cause of the  “N/A” and “test error” results and find a solution. We’ll be sure to share that information here in the blog once we do.

Justin

Posted in Battery life, benchmark, Benchmarking, Chrome OS, Chromebooks, CrXPRT, Google Chrome | Also tagged , , , ,

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2024   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x