Category: Future of performance evaluation

The name game

on June 5, 2014

In Something shiny, we discussed the leading contender in our search for new benchmark ideas, a benchmark tailored especially for the Chrome OS, and we’ve been looking at what workloads would make sense.

As we said, the ability to measure battery life would be useful. That’s not easy in the Chrome environment. We think we may be able to do it, but the Chromebook may have to be in developer mode. Even so, we can leverage what we’ve learned from BatteryXPRT to get a reliable estimate of battery life in less than a working day.

Measuring performance, however, is a must. We’ve been looking at the existing WebXPRT workloads as well as other applications, such as education apps, online games, HD video playback, music playback, and more. We’re also looking for areas where using native client execution makes sense, such as higher-resolution photo editing.

In addition, we’re thinking about what we might call this benchmark. ChromeXPRT would be obvious, but probably wouldn’t pass Google’s naming restrictions.

Do you have ideas for the benchmark’s name? Are there Chrome-based benchmark workloads you’d love to see? Let us know at BenchmarkXPRTsupport@principledtechnologies.com!

Eric

Comment on this post in the forums

Posted in Chrome OS, Collaborative benchmark development, Future of performance evaluation, WebXPRT |

Stronger, faster, and definitely better

By Eric Hale

on May 29, 2014

This week we released HDXPRT 2014 Community Preview 1 (CP1) to BenchmarkXPRT Development Community members. For those community members who’ve been around since the beginning, you’ll notice how much HDXPRT has changed. We’ve trimmed down HDXPRT without reducing the value of the test. While HDXPRT 2012 required multiple installation DVDs, HDXPRT 2014 CP1 is available for download in the member’s area. In addition, it took HDXPRT 2012 at least five hours to complete the recommended three test iterations. HDXPRT 2014 is much faster – the total time required for install and three test iterations is less than two hours!

Like all versions of HDXPRT, CP1 uses real-world media applications such as Adobe® Photoshop® Elements 12, Apple® iTunes®11.1, CyberLink MediaEspresso 6.7, and others to perform common consumer tasks like editing photos, converting videos, and editing music files.

If you’re a community member, give CP1 a try. If you have questions about HDXPRT 2014 or any of the XPRTs, please contact us at BenchmarkXPRTsupport@principledtechnologies.com. If you’d like to be a part of future XPRT development and community previews, now’s a great time to join!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Future of performance evaluation, HDXPRT 2012, HDXPRT 2014 |

Something shiny

By Eric Hale

on May 22, 2014

As I mentioned in Looking for the next big thing, we’ve received some ideas for benchmarks recently. While there were several good ideas, the one that rose to the top of the list was a benchmark for Chrome-based devices.

For the past year or so, Chromebooks have been the fastest-growing segment of the PC market, so there’s a lot of interest. Given the rising profile of Chromebooks, a Chrome OS benchmark seems like a good fit with the existing XPRTs.

We’ve just started batting around ideas about what a Chrome OS benchmark might look like. Because the browser is the interface, WebXPRT is the natural place to start. However, WebXPRT runs on all platforms. We would like a Chrome benchmark to consider Chrome‑specific technologies like Google Native Client. If possible, it would also be great to get the battery life as well as performance.

Those are a few of our ideas. Do you have ideas about what you would like to see in a Chrome OS benchmark? If so, what would you consider most important for it to include? Let us know at benchmarkxprtsupport@principledtechnologies.com!

And remember, we’re always looking for new ideas. If there’s a benchmark you’d love to see, let us know.

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, Future of performance evaluation, Google Chrome, WebXPRT, What makes a good benchmark? |

Looking for the next big thing

By Eric Hale

on April 17, 2014

We recently introduced a new Web form in the members’ area to make it easier for you to submit new benchmark ideas. We’ve already received some interesting suggestions:

A benchmark to assess performance, battery life, and Chrome-specific technologies on Chromebooks
A benchmark to evaluate camera features and photo quality on phones and tablets
A benchmark for measuring the performance of cloud services
A benchmark for measuring the performance and battery life of iOS-based devices

Are you interesting in seeing any of these? Or do you have an idea no one has mentioned yet? We know there’s more out there! We like finding new things to measure and new ways to measure them, so please don’t hesitate to share your ideas!

Also, remember that the comment period for BatteryXPRT 2014 for Android CP2 ends on Monday, April 21. CP2 is the first XPRT to feature a Simplified Chinese UI. Please send in your comments. We’ll be aiming for a BatteryXPRT general release soon.

By the way, if you have a language you’d like to see and you’re willing to help with the translation, we’d love to talk to you!

Let us know what you think about potential new benchmarks, language options, or anything else on your mind at BenchmarkXPRTSupport@principledtechnologies.com.

Eric

Comment on this post in the forums

Posted in BatteryXPRT 2014 for Android, Benchmarking, Chinese, Collaborative benchmark development, Community Preview, Future of performance evaluation, What makes a good benchmark? |

Golden tickets

By Bill Catchings

on January 9, 2014

We (Bill and Mark) are on our way home from CES. There were lots of cool things to see, from electric cars to health and fitness wearables to all manner of mobile devices. And, more, a whole lot more.

We enjoyed seeing many of those products, but that was not our primary mission at the show. Our main goal was to spread the word about the XPRT benchmarks. We did that by visiting multiple mobile-device members and giving out to many of them a very special golden ticket. Yes, we’re talking a physical, Willy Wonka-style golden ticket. The two-sided tickets look really cool.

One side invites folks to be heard by joining the BenchmarkXPRT community. The other offers them the opportunity to have PT test devices for free with all the applicable XPRT benchmarks. All a vendor has to do to get this free testing is send the device to PT. We hope to get many devices in-house and to provide a great many results on our Web sites.

We wore the new BenchmarkXPRT shirts as we walked the floor.

We will soon be sending one shirt—and one golden ticket—to each member of the community. Please make sure we have your latest mailing address so we can ship those to you.

-Bill & Mark

Comment on this post in the forums

Posted in BenchmarkXPRT development community, CES, Future of performance evaluation, Performance benchmarking, Principled Technologies |

Home sweet home

By Bill Catchings

on June 9, 2011

After a long set of flights back from Computex in Taipei, I’m finally home in North Carolina. Unfortunately, I’m still not quite sure what time zone I’m in!

While awake in the middle of the night, I’ve been thinking about some of the things I saw at Computex. While I was there, it seemed like a jumble of notebooks, power supplies, gaming rigs, motherboards, cases, Hello Kitty accessories, and some things that I still don’t quite know what they were. Many of the things I saw were not brand new, but it was my first chance to see them up close. Some of them were of technologies still on the horizon like Intel’s Ultrabook concept and Microsoft’s Windows 8. I also saw all sorts of combinations of phones, 4G, and other devices.

One thing that stood out to me were the number and variety of tablets. They were in a variety of sizes (and screen resolutions). There were quite a few vendors and some were ones I would not have suspected but was pleasantly surprised to encounter, like Viewsonic and Shuttle. The OS choices included Android, WebOS, and MeeGo. ASUS had a couple of interesting hybrid approaches such as the Eee Pad Transformer and the Padfone. The former is a 10.1-inch tablet that plugs into a keyboard. The Padfone is a smartphone that can plug into the back of a larger (10.1-inch) touch screen to act as a tablet.

All of these tablet choices, as well as the iPad that they all must be compete against, left me wondering how to choose between them. Some part of the choice comes down to the size and features. As always, however, performance plays a key role. My tolerance for waiting on a tablet device is even lower than it is for waiting on my PC. The problem is how to make valid comparisons across such a wide range of platforms. I’d love to hear from you what you think about performance testing on tablets. Is it useful? What are the best ways to accomplish it?

Finally, thanks to all the folks who came by and visited our suite at Computex. I enjoyed getting the chance to meet some of the members of the HDXPRT Development Community. And, hopefully, I convinced more folks to join.

Bill

Comment on this post in the forums

Posted in Future of performance evaluation, Performance testing on tablets |

Category: Future of performance evaluation

The name game

Stronger, faster, and definitely better

Something shiny

Looking for the next big thing

Golden tickets

Home sweet home

Check out the other XPRTs: