BenchmarkXPRT Blog banner

Category: Future of performance evaluation

More details to come

Eric Hale
By Eric Hale
on July 10, 2014

As we’ve been saying the past couple of months, we’re working on a benchmark for Chrome OS. The experimentation phase is winding down, and we are starting to shape the code into a useable benchmark. The design plan will leverage existing WebXPRT tests, of course. However, we’ve gone far beyond that. The benchmark will include video playback, 3D modeling via WebGL, and even an HTML5 game.  The test also uses Chrome OS’ native execution capability. The benchmark will actually use the Portable Native Client (PNaCl), as PNaCl is the recommended tool chain for native client. It also gives the benchmark the ability to run on more platforms.

As we mentioned before, we’re including a battery test as part of the new benchmark. So far, we haven’t found a way to remove the requirement to put the device in developer mode for the battery test.

Next week, we’ll publish a design document for the community to review. As always, the design document is based on the comments and suggestions we received combined with our own research and experimentation.

Eric

Comment on this post in the forums

Posted in Chrome OS, Future of performance evaluation, HTML5, Performance benchmarking, WebXPRT |

The name game

Eric Hale
By Eric Hale
on June 5, 2014

In Something shiny, we discussed the leading contender in our search for new benchmark ideas, a benchmark tailored especially for the Chrome OS, and we’ve been looking at what workloads would make sense.

As we said, the ability to measure battery life would be useful. That’s not easy in the Chrome environment. We think we may be able to do it, but the Chromebook may have to be in developer mode. Even so, we can leverage what we’ve learned from BatteryXPRT to get a reliable estimate of battery life in less than a working day.

Measuring performance, however, is a must. We’ve been looking at the existing WebXPRT workloads as well as other applications, such as education apps, online games, HD video playback, music playback, and more. We’re also looking for areas where using native client execution makes sense, such as higher-resolution photo editing.

In addition, we’re thinking about what we might call this benchmark. ChromeXPRT would be obvious, but probably wouldn’t pass Google’s naming restrictions.

Do you have ideas for the benchmark’s name? Are there Chrome-based benchmark workloads you’d love to see? Let us know at BenchmarkXPRTsupport@principledtechnologies.com!

Eric

Comment on this post in the forums

Posted in Chrome OS, Collaborative benchmark development, Future of performance evaluation, WebXPRT |

Stronger, faster, and definitely better

Eric Hale
By Eric Hale
on May 29, 2014

This week we released HDXPRT 2014 Community Preview 1 (CP1) to BenchmarkXPRT Development Community members. For those community members who’ve been around since the beginning, you’ll notice how much HDXPRT has changed. We’ve trimmed down HDXPRT without reducing the value of the test. While HDXPRT 2012 required multiple installation DVDs, HDXPRT 2014 CP1 is available for download in the member’s area. In addition, it took HDXPRT 2012 at least five hours to complete the recommended three test iterations. HDXPRT 2014 is much faster – the total time required for install and three test iterations is less than two hours!

Like all versions of HDXPRT, CP1 uses real-world media applications such as Adobe® Photoshop® Elements 12, Apple® iTunes®11.1, CyberLink MediaEspresso 6.7, and others to perform common consumer tasks like editing photos, converting videos, and editing music files.

If you’re a community member, give CP1 a try. If you have questions about HDXPRT 2014 or any of the XPRTs, please contact us at BenchmarkXPRTsupport@principledtechnologies.com. If you’d like to be a part of future XPRT development and community previews, now’s a great time to join!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Future of performance evaluation, HDXPRT 2012, HDXPRT 2014 |

Something shiny

Eric Hale
By Eric Hale
on May 22, 2014

As I mentioned in Looking for the next big thing, we’ve received some ideas for benchmarks recently. While there were several good ideas, the one that rose to the top of the list was a benchmark for Chrome-based devices.

For the past year or so, Chromebooks have been the fastest-growing segment of the PC market, so there’s a lot of interest. Given the rising profile of Chromebooks, a Chrome OS benchmark seems like a good fit with the existing XPRTs.

We’ve just started batting around ideas about what a Chrome OS benchmark might look like. Because the browser is the interface, WebXPRT is the natural place to start. However, WebXPRT runs on all platforms. We would like a Chrome benchmark to consider Chrome‑specific technologies like Google Native Client. If possible, it would also be great to get the battery life as well as performance.

Those are a few of our ideas. Do you have ideas about what you would like to see in a Chrome OS benchmark? If so, what would you consider most important for it to include? Let us know at benchmarkxprtsupport@principledtechnologies.com!

And remember, we’re always looking for new ideas. If there’s a benchmark you’d love to see, let us know.

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, Future of performance evaluation, Google Chrome, WebXPRT, What makes a good benchmark? |

Looking for the next big thing

Eric Hale
By Eric Hale
on April 17, 2014

We recently introduced a new Web form in the members’ area to make it easier for you to submit new benchmark ideas. We’ve already received some interesting suggestions:

  • A benchmark to assess performance, battery life, and Chrome-specific technologies on Chromebooks
  • A benchmark to evaluate camera features and photo quality on phones and tablets
  • A benchmark for measuring the performance of cloud services
  • A benchmark for measuring the performance and battery life of iOS-based devices

Are you interesting in seeing any of these? Or do you have an idea no one has mentioned yet? We know there’s more out there! We like finding new things to measure and new ways to measure them, so please don’t hesitate to share your ideas!

Also, remember that the comment period for BatteryXPRT 2014 for Android CP2 ends on Monday, April 21. CP2 is the first XPRT to feature a Simplified Chinese UI. Please send in your comments. We’ll be aiming for a BatteryXPRT general release soon.

By the way, if you have a language you’d like to see and you’re willing to help with the translation, we’d love to talk to you!

Let us know what you think about potential new benchmarks, language options, or anything else on your mind at BenchmarkXPRTSupport@principledtechnologies.com.

Eric

Comment on this post in the forums

Posted in BatteryXPRT 2014 for Android, Benchmarking, Chinese, Collaborative benchmark development, Community Preview, Future of performance evaluation, What makes a good benchmark? |

Golden tickets

Bill Catchings
By Bill Catchings
on January 9, 2014

We (Bill and Mark) are on our way home from CES. There were lots of cool things to see, from electric cars to health and fitness wearables to all manner of mobile devices. And, more, a whole lot more.

We enjoyed seeing many of those products, but that was not our primary mission at the show. Our main goal was to spread the word about the XPRT benchmarks. We did that by visiting multiple mobile-device members and giving out to many of them a very special golden ticket. Yes, we’re talking a physical, Willy Wonka-style golden ticket. The two-sided tickets look really cool.

One side invites folks to be heard by joining the BenchmarkXPRT community. The other offers them the opportunity to have PT test devices for free with all the applicable XPRT benchmarks. All a vendor has to do to get this free testing is send the device to PT. We hope to get many devices in-house and to provide a great many results on our Web sites.

We wore the new BenchmarkXPRT shirts as we walked the floor.

We will soon be sending one shirt—and one golden ticket—to each member of the community. Please make sure we have your latest mailing address so we can ship those to you.

-Bill & Mark

Comment on this post in the forums

Posted in BenchmarkXPRT development community, CES, Future of performance evaluation, Performance benchmarking, Principled Technologies |

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2024   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x