Category: Benchmarking

Sounds easy, but…

on December 5, 2013

Sounds easy, but…

In Endurance, Bill said that we were going to be investigating battery life testing. He also discussed some of the issues that make battery testing difficult to do well. Finally, he explained why we were looking at MobileXPRT as the basis for the first version of the battery life test.

Over the last couple of months, we have been experimenting with a number of different approaches to battery testing. We now think that we have enough empirical data that we can make a proposal. We are working on that now. It should be available to community members in the next couple of weeks.

We hope you’ll look at the proposal and let us know what you think. Your input is an essential part of developing a really great test. If you’re not a member of the community, it’s easy to join.

In other news, we’re going to CES and would love to talk with you. If you’d like to chat, send an e-mail to benchmarkxprtsupport@principledtechnologies.com.

Eric

Comment on this post in the forums

Posted in Benchmarking, Collaborative benchmark development, MobileXPRT, What makes a good benchmark? |

Diversity

By Eric Hale

on November 7, 2013

Fall is beautiful in North Carolina. The temperature is dropping. The leaves are changing color, making the hills scarlet and orange. And, of course, the stores have been decorated for Christmas since Halloween.

As we head into the biggest shopping season of the year, it’s a great time to be getting XPRT results from the hottest devices. In the last few weeks, we’ve published results from

tablets such as the Apple iPad Air, Google Nexus 7 2, and both the Microsoft Surface 2 and Microsoft Surface Pro 2
phones such as the Apple iPhone 5c, Apple iPhone 5s, and LG G2
devices you might not have expected, such as the Amazon Kindle Fire HDX, Barnes and Noble Nook HD+, and NVIDIA Shield

The diversity of devices is nice to see. The results come from PT testing, the press, and benchmark users. Note that you don’t have to be a community member to submit results. The person who submitted the MobileXPRT Nook HD+ results was not a member. If you’ve tested something interesting, send the results on!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, MobileXPRT, Performance testing on tablets, Touch-based computing, Touch-based devices, TouchXPRT, TouchXPRT results |

Broadening our appeal

By Eric Hale

on October 31, 2013

As I mentioned last week, we’ve asked the PT design team to help improve the XPRT benchmarks. I’m learning a lot working with them. As someone who’s been involved with benchmarking a long time, it can be a shock to realize that there are people who think “ms” is a magazine, “geomean” has something to do with the environment, and “UX” sounds like it would be a great name for a band. But the fact is that most consumers don’t need to know any of these terms, so our benchmarks shouldn’t rely on them, either.

This collaboration already paying off. The PT design team rewrote the MobileXPRT FAQ, making it much more extensive, accessible, and fun to read. We think the new FAQ is greatly improved, and it’s certainly more informative. We’ll be upgrading the FAQs for the other benchmarks in the near future.

Our efforts are going far beyond FAQs. Data presentation, graphics, the basic UI design philosophy—everything is on the table. Let us know what you think by emailing benchmarkxprtsupport@principledtechnolgies.com or by posting on the forums.

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Let us know your thoughts, MobileXPRT |

The Microsoft Surface 2

By Eric Hale

on October 24, 2013

As soon as the Microsoft Surface 2 became available, we got one and have been putting it through its paces. Of course, we ran WebXPRT and TouchXPRT. The results are on the TouchXPRT and WebXPRT sites, but I’ll repeat them here along with the results for its predecessor, the Microsoft Surface RT.

	TouchXPRT	WebXPRT
Surface RT	98	167
Surface 2	284	324

TouchXPRT shows the Surface 2 to be almost three times faster than the Microsoft Surface RT, while WebXPRT shows it to be almost twice as fast.

Why the difference? The most obvious explanation is that WebXPRT depends on the browser and its implementations of JavaScript and HTML5. TouchXPRT relies less on additional software and seems to take better advantage of the underlying hardware.

While we have yet to test the Intel Core i5-based Microsoft Surface Pro 2 ourselves, others have been doing so. Interestingly, Anandtech’s review of the Surface Pro 2 included WebXPRT results from both Chrome and IE. The Chrome result was over 30 percent higher than the IE result: 1,260 vs. 960. Unfortunately, Google has not made Chrome available for the ARM-based Surface 2, so we were not able to make that comparison.

As always, please let us know any results you get on any new hardware so we can get as many results as possible in our result databases. There are lots of new products coming out in the next few weeks and we’d love your help in getting results for as many of them as possible. Thanks!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Performance testing on tablets, Touch-based computing, Touch-based devices, TouchXPRT, TouchXPRT results, WebXPRT, Windows 8.1 |

Sprucing things up

By Eric Hale

on October 17, 2013

A major goal of the BenchmarkXPRT benchmarks is to be useful to the people who buy consumer devices, most of whom do not have experience running benchmarks. As these people have started discovering the benchmarks, we’ve been learning a lot about how to make the benchmarks clearer and easier to use.

We’re working on a lot of things: making the data clearer and easier to understand, giving better feedback about what the benchmark is doing, making the controls clearer and easier to find, making the documentation more accessible, and more. We also need to make the UIs of the benchmarks more unified, so that the benchmarks appear to be the members of the same family.

Because we take this effort very seriously, we are involving the PT design team in a more integral way. These are the same people who spearheaded the Web site redesign and produced the BenchmarkXPRT videos. (See their most recent video here.)

We’ll keep you informed about the design process as we go. If you have complaints or suggestions, it’s not too late. Please post to the forums or send an email to benchmarkxprtsupport@principledtechnologies.com.

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Principled Technologies |

Look how we’ve grown!

By Eric Hale

on October 10, 2013

We talk a lot about the community model, but perhaps not enough about the community itself. I was looking at our growth over the past few months, and struck by how far we have come this year. We don’t reveal our members’ personal information, but I would like to share some statistics with you.

The community has grown by over 40 percent in 2013. Members are affiliated with 49 organizations, representing chip makers, OEMs, software vendors, and the press. Some members work outside technology, in fields such as engineering, finance, and communications.

Back in April, we blogged about setting up a second WebXPRT host to help with download speeds in China, so it’s no surprise that the community has also spread across the world. Because we don’t require detailed information from community members, I don’t know how many countries are represented. However, the members with whom I’ve had personal contact come from at least four continents.

Thanks to all our community members for making it a great year so far. If you’ve not joined yet, now is a great time. Remember that members see the previews of the benchmarks, and the 2014 versions are coming.

Here’s looking forward to greater growth in the future.

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, WebXPRT |

Category: Benchmarking

Sounds easy, but…

Diversity

Broadening our appeal

The Microsoft Surface 2

Sprucing things up

Look how we’ve grown!

Check out the other XPRTs: