Category: What makes a good benchmark?

Upcoming experiments

on November 6, 2014

Next week, we’ll be releasing the design overview for WebXPRT 2015. WebXPRT 2013 has been an enormous success, having been run tens of thousands of times.

One of the big improvements we are considering for WebXPRT 2015 is adding experimental tests. A big reason for WebXPRT’s success is that it runs on almost every Web-enabled device. We consider it essential to preserve this broad compatibility. However, there are interesting Web technologies that simply are not available on all devices.

Our proposal is to add experimental tests to WebXPRT. These tests would be optional and would not be included in the Overall score, so WebXPRT would still be able to compare the performance of widely different devices. We are looking at technologies such as Web Workers, WebGL, and pre-compiled JavaScript (asm.js).

In addition to adding experimental tests, we are looking at ways to improve the UI, add automation, add new tests, update old tests, and more!

If you are a community member, you’ll get a notice when the overview is available. We will definitely want to know what you think! If you are not a member, it’s a great time to join.

If you have any thoughts on these ideas, or have ideas of your own, please let us know!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Performance benchmarking, WebXPRT, WebXPRT 2015, What makes a good benchmark? |

What do you think when you hear “Chromebook”?

By Justin Greene

on September 11, 2014

We’ve been thinking a lot about Chromebooks while doing all of our testing in preparation for the CrXPRT Community Preview. In both the models we’re testing and the ones announced in the press, we’ve seen just how much the Chromebook market is changing. Some folks even claim that Chromebook sales made up 35 percent of US commercial laptop sales in the first half of 2014. What’s even more interesting to us is the wide variety of Chromebooks on the market.

Choosing between Chromebooks these days is becoming more complicated than it used to be. There’s a greater range of hardware choices, and those choices can have a direct impact on performance and battery life. Some Chromebooks offer local storage up to 320 GB, touch screens, and 4G/LTE connectivity. Prices range widely, from $199 to $1,499. Even seemingly comparable systems can perform much differently when put to the test. For instance, we recently tested two Chromebooks separated by only $50 in price, but over 5 hours of estimated battery life!

Whether a consumer’s ultimate purchasing decision is based on price, specs, or a combination of factors, there are few things more valuable to buyers than reliable facts about performance and battery life. Benchmarking is ultimately about gaining useful data for decision making, and that’s why we’re excited about the value that CrXPRT will bring to the Chromebook discussion!

Justin

Comment on this post in the forums

Posted in Chrome OS, CrXPRT, Google Chrome, Performance of computing devices, What makes a good benchmark? |

Time to get creative

By Justin Greene

on September 4, 2014

The CrXPRT Community Preview is right around the corner, and there’s no sign of things slowing down. We’re exploring new opportunities on a number of fronts, and we’d love to hear what you think! We’re considering possible changes to WebXPRT and MobileXPRT, and since the mobile device market is changing all the time, we’re looking for the next great benchmark opportunity. In both cases, the development community is a rich source of ideas, so we’d like to tap into it one more time.

A while back, we added a new Web form in the members’ area for submitting benchmark ideas. Some of the ideas we have so far include:

A benchmark to evaluate camera features and photo quality on phones and tablets
A benchmark for measuring the performance of cloud services
A benchmark for measuring the performance and battery life of iOS-based devices

So, what would you like to see? Any of these, or do you have ideas we haven’t mentioned? Also, we’d love to hear your feedback on ways we can improve, both with the XPRTs themselves and with community life. Either way, send a message to BenchmarkXPRTSupport@principledtechnologies.com and let us know what you think!

Justin

Comment on this post in the forums

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, CrXPRT, MobileXPRT, WebXPRT, What makes a good benchmark? |

Speaking the same language

By Eric Hale

on August 7, 2014

We count on our community members for so much: benchmark ideas, critiquing the benchmark designs, and testing the community previews. Community members with programming skills can vet the source code and submit code for inclusion in the benchmarks.

We love getting code from our members. However, people have widely differing opinions about what constitutes well-documented code. A lot of it comes down to taste, but it’s easier to read code when there are common conventions. So, we’ve put together a very brief description of some conventions that would make it easier to read your code.

Because the XPRT benchmarks are written in a number of languages, we don’t discuss the particulars of coding style in detail. We know that you know the best practices for your language of choice. However, when we’re reading code in C, C++, C#, Java, JavaScript, HTML5, XML, and more, it helps to have some points of reference.

So, check it out and let us know what you think. If you have code to share, please post on the forums or send us a message at BenchmarkXPRTsupport@principledtechnologies.com. We can’t wait to see what you’ve come up with!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Source code, What makes a good benchmark? |

Testing the waters

By Eric Hale

on June 19, 2014

A couple of weeks ago, we talked about some of our ideas for a new XPRT designed for Google’s Chrome OS. We’ve been working with some of these ideas and, while we’re still in the experimental stage, things look promising so far.

As we mentioned in the earlier blog, we’re trying WebXPRT as a base for the performance part of the test. So far, the performance component is working well. In addition to modified WebXPRT tests, we’re also trying some things that are not part of the WebXPRT 2013 workload.

We’ve been able to get battery life, but it’s been challenging and we haven’t found a way to avoid using Chrome’s Developer Mode. Accessing Developer Mode in Chrome can be tricky and requires different steps for each hardware manufacturer. We’re hoping to find ways to make battery life testing easier.

I’ve been vague about the tests because they’re likely to change over the next few weeks. We’re experimenting with both browser-based and Native Client-based performance tasks. As they firm up, I’ll be able to share more information.

Challenges aside, we’re excited about this new benchmark, and committed to making it as effective as possible. We’d still love feedback on a name, so feel free to contact us at BenchmarkXPRTsupport@principledtechnologies.com with your ideas.

Eric

Comment on this post in the forums

Posted in Battery life, Chrome OS, Collaborative benchmark development, Performance benchmarking, What makes a good benchmark? |

Something shiny

By Eric Hale

on May 22, 2014

As I mentioned in Looking for the next big thing, we’ve received some ideas for benchmarks recently. While there were several good ideas, the one that rose to the top of the list was a benchmark for Chrome-based devices.

For the past year or so, Chromebooks have been the fastest-growing segment of the PC market, so there’s a lot of interest. Given the rising profile of Chromebooks, a Chrome OS benchmark seems like a good fit with the existing XPRTs.

We’ve just started batting around ideas about what a Chrome OS benchmark might look like. Because the browser is the interface, WebXPRT is the natural place to start. However, WebXPRT runs on all platforms. We would like a Chrome benchmark to consider Chrome‑specific technologies like Google Native Client. If possible, it would also be great to get the battery life as well as performance.

Those are a few of our ideas. Do you have ideas about what you would like to see in a Chrome OS benchmark? If so, what would you consider most important for it to include? Let us know at benchmarkxprtsupport@principledtechnologies.com!

And remember, we’re always looking for new ideas. If there’s a benchmark you’d love to see, let us know.

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, Future of performance evaluation, Google Chrome, WebXPRT, What makes a good benchmark? |

Category: What makes a good benchmark?

Upcoming experiments

What do you think when you hear “Chromebook”?

Time to get creative

Speaking the same language

Testing the waters

Something shiny

Check out the other XPRTs: