Category: WebXPRT

Time to get creative

on September 4, 2014

The CrXPRT Community Preview is right around the corner, and there’s no sign of things slowing down. We’re exploring new opportunities on a number of fronts, and we’d love to hear what you think! We’re considering possible changes to WebXPRT and MobileXPRT, and since the mobile device market is changing all the time, we’re looking for the next great benchmark opportunity. In both cases, the development community is a rich source of ideas, so we’d like to tap into it one more time.

A while back, we added a new Web form in the members’ area for submitting benchmark ideas. Some of the ideas we have so far include:

A benchmark to evaluate camera features and photo quality on phones and tablets
A benchmark for measuring the performance of cloud services
A benchmark for measuring the performance and battery life of iOS-based devices

So, what would you like to see? Any of these, or do you have ideas we haven’t mentioned? Also, we’d love to hear your feedback on ways we can improve, both with the XPRTs themselves and with community life. Either way, send a message to BenchmarkXPRTSupport@principledtechnologies.com and let us know what you think!

Justin

Comment on this post in the forums

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, CrXPRT, MobileXPRT, WebXPRT, What makes a good benchmark? |

More details to come

By Eric Hale

on July 10, 2014

As we’ve been saying the past couple of months, we’re working on a benchmark for Chrome OS. The experimentation phase is winding down, and we are starting to shape the code into a useable benchmark. The design plan will leverage existing WebXPRT tests, of course. However, we’ve gone far beyond that. The benchmark will include video playback, 3D modeling via WebGL, and even an HTML5 game. The test also uses Chrome OS’ native execution capability. The benchmark will actually use the Portable Native Client (PNaCl), as PNaCl is the recommended tool chain for native client. It also gives the benchmark the ability to run on more platforms.

As we mentioned before, we’re including a battery test as part of the new benchmark. So far, we haven’t found a way to remove the requirement to put the device in developer mode for the battery test.

Next week, we’ll publish a design document for the community to review. As always, the design document is based on the comments and suggestions we received combined with our own research and experimentation.

Eric

Comment on this post in the forums

Posted in Chrome OS, Future of performance evaluation, HTML5, Performance benchmarking, WebXPRT |

It makes a difference

By Eric Hale

on July 3, 2014

Ars Technica reported this week that they tested the developer preview of Android L and saw a whopping 36 percent improvement in battery life! Google made improving battery life a priority, and it sounds like they are succeeding. I can’t wait to test Android L with BatteryXPRT.

This is a spectacular example of how a change in software can change benchmark results, but it’s hardly unique. I’ve written before about how background activity on a phone depressed my friend’s WebXPRT scores. AnandTech used both IE 11 and Chrome 30 to test the Surface Pro 2 with a variety of benchmarks, including WebXPRT, SunSpider, Octane, Browsermark, and others. Browser choice had a noticeable impact on results – about a 40 percent difference for WebXPRT and a 76 percent difference for SunSpider!

People are generally pretty aware that changing the hardware changes performance. However, sometimes they lose track of software differences. When you compare scores, it’s not always possible to keep all the variables the same, but it’s crucial to know what the differences are.

In other BenchmarkXPRT news, we’re making some final adjustments to HDXPRT 2014, and the general release is just around the corner.

Eric

Comment on this post in the forums

Posted in Android, Battery life, HDXPRT 2014, Performance benchmarking, Performance testing on tablets, WebXPRT |

The name game

By Eric Hale

on June 5, 2014

In Something shiny, we discussed the leading contender in our search for new benchmark ideas, a benchmark tailored especially for the Chrome OS, and we’ve been looking at what workloads would make sense.

As we said, the ability to measure battery life would be useful. That’s not easy in the Chrome environment. We think we may be able to do it, but the Chromebook may have to be in developer mode. Even so, we can leverage what we’ve learned from BatteryXPRT to get a reliable estimate of battery life in less than a working day.

Measuring performance, however, is a must. We’ve been looking at the existing WebXPRT workloads as well as other applications, such as education apps, online games, HD video playback, music playback, and more. We’re also looking for areas where using native client execution makes sense, such as higher-resolution photo editing.

In addition, we’re thinking about what we might call this benchmark. ChromeXPRT would be obvious, but probably wouldn’t pass Google’s naming restrictions.

Do you have ideas for the benchmark’s name? Are there Chrome-based benchmark workloads you’d love to see? Let us know at BenchmarkXPRTsupport@principledtechnologies.com!

Eric

Comment on this post in the forums

Posted in Chrome OS, Collaborative benchmark development, Future of performance evaluation, WebXPRT |

Something shiny

By Eric Hale

on May 22, 2014

As I mentioned in Looking for the next big thing, we’ve received some ideas for benchmarks recently. While there were several good ideas, the one that rose to the top of the list was a benchmark for Chrome-based devices.

For the past year or so, Chromebooks have been the fastest-growing segment of the PC market, so there’s a lot of interest. Given the rising profile of Chromebooks, a Chrome OS benchmark seems like a good fit with the existing XPRTs.

We’ve just started batting around ideas about what a Chrome OS benchmark might look like. Because the browser is the interface, WebXPRT is the natural place to start. However, WebXPRT runs on all platforms. We would like a Chrome benchmark to consider Chrome‑specific technologies like Google Native Client. If possible, it would also be great to get the battery life as well as performance.

Those are a few of our ideas. Do you have ideas about what you would like to see in a Chrome OS benchmark? If so, what would you consider most important for it to include? Let us know at benchmarkxprtsupport@principledtechnologies.com!

And remember, we’re always looking for new ideas. If there’s a benchmark you’d love to see, let us know.

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, Future of performance evaluation, Google Chrome, WebXPRT, What makes a good benchmark? |

More grease, more tracks

By Eric Hale

on March 27, 2014

We’re always looking for ways to make it easier for people to use the benchmarks and interact with the community. I wrote about a new form for suggesting new benchmarks last week. However, we are not stopping there.

Although the community previews for TouchXPRT 2014 and BatteryXPRT 2014 for Android have been out for a few weeks, we’ve continued to improve them before their general releases. For example, we mentioned in TouchXPRT CP1 that we would continue making changes to the UI.

Taking a cue from WebXPRT 2013, the upcoming general releases of TouchXPRT 2014 and BatteryXPRT 2014 for Android will have a button in the UI for submitting results. This will make it easier for you to submit your results to be approved for publication on the PT Web site.

Although there’s not much time left before the general releases of these benchmarks, it’s not too late to send in your ideas. Even if we can’t use your suggestions in this release, we may be able to use them in the future. Let us know what you think by posting to the forum or emailing us at BenchmarkXPRTsupport@principledtechnologies.com.

Eric

Comment on this post in the forums

Posted in BatteryXPRT 2014 for Android, Collaborative benchmark development, Community Preview, Let us know your thoughts, TouchXPRT, TouchXPRT 2014, WebXPRT |

Category: WebXPRT

Time to get creative

More details to come

It makes a difference

The name game

Something shiny

More grease, more tracks

Check out the other XPRTs: