Last month, we discussed a potential fix for the error that was preventing CrXPRT 2 testers from successfully completing battery life tests on systems running Chrome v89.x and later. Since then, we’ve been testing an updated, unpublished version of the app package across several Chromebook models to ensure that the new build is stable and produces consistent results. We’re happy to report that our testing was successful, and we’ve published the new CrXPRT build (v1.2.0.0) in the Chrome Web Store and it is live as of 12:45 PM EDT today.
Note
that it might take some time for the update to appear on your Chromebook and,
once it does, you might have to manually approve the update notice.
Neither
the tests nor the method of calculating the overall score and battery-life
score in this new build have changed, so results are comparable with previous
CrXPRT 2 results.
We appreciate everyone’s patience while we found a solution to the error. If you have any questions or comments about the CrXPRT 2 battery life test, please feel free to contact us!
Device reviews in publications
such as AnandTech, Notebookcheck, and PCMag, among many others, often feature
WebXPRT test results, and we appreciate the many members of the tech press that
use WebXPRT. As we move forward with the WebXPRT 4 development process, we’re especially
interested in learning what longtime users would like to see in a new version
of the benchmark.
In previous posts,
we’ve asked people to weigh in on the potential addition of a WebAssembly workload or a battery life test. We’d also like to ask experienced testers some other
test-related questions. To that end, this week we’ll be sending a WebXPRT 4
survey directly to members of the tech press who frequently publish WebXPRT
test results.
Regardless of whether you are a member of the tech press, we invite you to participate by sending your answers to any or all the questions below to benchmarkxprtsupport@principledtechnologies.com. We ask you to do so by the end of May.
Do you think WebXPRT 3’s selection of workload scenarios is representative of modern web tasks?
How do you think WebXPRT compares to other common browser-based benchmarks, such as JetStream, Speedometer, and Octane?
Are there web technologies that you’d like us to include in additional workloads?
Are you happy with the WebXPRT 3 user interface? If not, what UI changes would you like to see?
Are there any aspects of WebXPRT 2015 that we changed in WebXPRT 3 that you’d like to see us change back?
Have you ever experienced significant connection issues when testing with WebXPRT?
Given the array of workloads, do you think the WebXPRT runtime is reasonable? Would you mind if the average runtime were a bit longer?
Are there any other aspects of WebXPRT 3 that you’d like to see us change?
If you’d like to discuss any topics
that we did not cover in the questions above, please feel free to include additional
comments in your response. We look forward to hearing your thoughts!
A few weeks ago, we discussed error messages that a tester received when starting up CrXPRT 2
after a battery life test. CrXPRT 2 battery life tests require a full battery
rundown, after which the tester plugs in the Chromebook, turns it on, opens the
CrXPRT 2 app, and sees the test results. In the reported cases, the tester
opened the app after a battery life test that seemed successful, but saw “N/A” or
“test error” messages instead of the results they expected.
During discussions about the end-of-test
system environment, we realized that some testers might be unclear about how to
tell that the battery has fully run down. During the system idle portion of
CrXPRT 2 battery life test iterations, the Chromebook screen turns black and a
small cursor appears somewhere on the screen to let testers know the test is
still in progress. We believe that some testers, seeing the black screen but not
the cursor, believe the system has shut down. Restarting CrXPRT 2 before the
battery life test is complete could explain some of the “N/A” or “test error” messages
users have reported.
If you see a black screen without
a cursor, you can check to see whether the test is complete by looking for the
small system power indicator light on the side or top of most Chromebooks.
These are usually red, orange, or green, but if a light of any color is lit, the
test is still underway. When the light goes out, the test has ended. You can
plug the system in and power it on to see results.
Note that some Chromebooks provide
low-battery warnings onscreen. During CrXPRT 2 battery life runs, testers
should ignore these.
We hope this clears up any
confusion about how to know when a CrXPRT 2 battery life test has ended. If
you’ve received repeated “N/A” or “test error” messages during your CrXPRT 2
testing and the information above does not help, please let us know!
It’s
been two weeks since the CrXPRT 2
public release, and we’re happy to see widespread interest in the test right
out of the gate!
This
week, we received a couple of questions about its battery life test from
Melissa Riofrio at PCWorld.
First, she asked for clarification about the relationship between the rundown
time and the 30-minute increments that appear in the iteration details table
for each battery life run. Second, she asked what could be causing her to get
“N/A” and “test error” battery life results at the end of what appeared to be
successful tests. Both topics may be of interest to other CrXPRT 2 testers, so
we’ve decided to address them here in the blog and invite our readers to
provide any relevant feedback.
Rundown
time vs. elapsed time
When
you’re viewing previous CrXPRT 2 test results and click the Details link for a
specific battery life test run, a window displaying additional test information
appears (the screenshot below shows an example). The window first provides
performance details, then presents a table with several data points for each iteration.
The data point in the far-right column, elapsed time, can be slightly confusing. Each test iteration runs for 30 minutes, and this column provides a cumulative total of these 30-minute increments. In some instances, these totals accurately reflect the actual time elapsed from the time that testing begins. However, if the test system shuts down for some reason before running the entire iteration, this table will still show the entire 30 minutes allotted for the that iteration. In these cases, the cumulative elapsed time value in the far-right column will not match the rundown time that the test reports for the system’s battery life. For that reason, testers should always consider rundown time as the definitive value for battery life.
“N/A” and “test error” battery life results after
apparently successful tests
We’re
actively investigating this issue at present. We’ve tested a wide range of
Chromebooks, both old and new, on several versions of Chrome OS, including the
latest versions, and have been unable to reproduce the problem. Have you witnessed
this behavior at the end of a CrXPRT 2 battery life test? If so, we’d love to get
more information from you about the system under test and your testing
procedures, so please contact us.
We’re
grateful to Melissa for raising these questions, and we appreciate everyone’s feedback
on CrXPRT 2. Hopefully, we’ll soon be able to determine the cause of the “N/A” and “test error” results and find a
solution. We’ll be sure to share that information here in the blog once we do.
We’re
excited to announce that CrXPRT 2 is now available to the public! Testers can
install the app on Chromebooks by using this link
to the Chrome Web Store listing, or by clicking the Chrome Web Store button on CrXPRT.com.
For
those who may have missed our previous discussions about what’s new with CrXPRT
2, here is a recap of the key differences between CrXPRT 2015 and CrXPRT 2:
CrXPRT 2 has an all-new UI, with a focus on intuitive navigation.
The CrXPRT 2 performance test includes six of the seven workloads in CrXPRT 2015. Newer versions of Chrome can’t run the Photo Collage workload without a workaround, so we removed it from CrXPRT 2.
We updated the image resolutions and sizes for the Photo Effects and Face Detection workloads.
The battery life test now requires a full rundown, so the length of battery life tests will vary according to the battery life of the systems under test.
CrXPRT 2 no longer requires testers to enter luminance and audio measurements for battery life tests.
We added a second video playback segment to each battery life iteration.
As
we’ve noted before, CrXPRT 2 overall performance test scores and battery life
measurements are not comparable to CrXPRT 2015 scores. For testers that need to
access CrXPRT 2015 for legacy comparison testing, we will continue to make
CrXPRT 2015 available via a link on CrXPRT.com.
We appreciate everyone’s input during the CrXPRT 2 development process. If you have any further questions about CrXPRT 2, please let us know!
Today, we want to share quick
updates on a few XPRT topics.
In case you missed yesterday’s announcement, the CrXPRT 2 Community Preview (CP) is now available. BenchmarkXPRT Development Community members can access the preview using a direct link we’ve posted on the CrXPRT tab in the XPRT Members’ Area (login required). This tab also provides a link to the CrXPRT 2 CP user manual. You can find a summary of what’s new with CrXPRT 2 in last week’s blog. During the preview period, we allow testers to publish CP test scores. Note that CrXPRT 2 overall performance test scores and battery life measurements are not comparable to those from CrXPRT 2015.
We’ll soon be publishing our
first AIXPRT whitepaper,
Introduction to AIXPRT. It will summarize the AIXPRT toolkits and
workloads; how to adjust test parameters such as batch size, levels of
precision, and concurrent instances; how to use alternate test configuration
files; and how to understand test results. When the paper is available, we’ll
post it on the XPRT white papers
page and make an announcement here in the blog.
Finally, in response to decreased downloads and usage of BatteryXPRT, we have ended support for the benchmark. We’re always monitoring usage of the XPRTs so that we can better direct our resources to the current needs of users. We’ve removed BatteryXPRT from the Google Play Store, but it is still available for download on BatteryXPRT.com.
If you have any questions about CrXPRT
2, AIXPRT, or BatteryXPRT, please let us know!
Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.
