Author Archives: Eric Hale

Sounds easy, but…

on December 5, 2013

Sounds easy, but…

In Endurance, Bill said that we were going to be investigating battery life testing. He also discussed some of the issues that make battery testing difficult to do well. Finally, he explained why we were looking at MobileXPRT as the basis for the first version of the battery life test.

Over the last couple of months, we have been experimenting with a number of different approaches to battery testing. We now think that we have enough empirical data that we can make a proposal. We are working on that now. It should be available to community members in the next couple of weeks.

We hope you’ll look at the proposal and let us know what you think. Your input is an essential part of developing a really great test. If you’re not a member of the community, it’s easy to join.

In other news, we’re going to CES and would love to talk with you. If you’d like to chat, send an e-mail to benchmarkxprtsupport@principledtechnologies.com.

Eric

Comment on this post in the forums

Posted in Benchmarking, Collaborative benchmark development, MobileXPRT, What makes a good benchmark? |

TouchXPRT 2014

By Eric Hale

on November 21, 2013

The code for TouchXPRT 2014 is pretty far along, and we plan to move into concentrated testing in the next week or so.

A lot of work has gone into the next version of TouchXPRT and it shows. We’re making changes to the content, including larger, higher resolution photos. The revised tests are not only more demanding than the current versions, but they also introduce new types of work. For example, the export to MP3 test now mixes several music tracks before exporting the result and the photo tests include merging the photos.

The tests are also more visual than before. You’ll see the photo merge, the music mixing, and so on. We’ve really tried to make the tests tell a story, and we think you’ll recognize the way you use your device when you see them.

We’re also fixing some issues on the UI side. TouchXPRT will include a proper results browser, and have an in-UI way to submit your results. Also, the Run all button, which is on the app bar, will be visible on the main screen.

I’ll have more to say over the coming weeks. We hope to have the community preview ready to go in December. You will definitely want to look at this one!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, TouchXPRT |

It’s the same phone

By Eric Hale

on November 14, 2013

A friend of mine thought there was something wrong with her phone. The reason? Her WebXPRT score was a lot lower than the score we published on the WebXPRT Web site.

From the frequency with which she posts on Facebook, I guessed she had the Facebook app running on her phone. So I asked if she had installed any apps on her phone since she bought it. She said “a few,” and I suggested we try turning some of them off.

Sometimes “a few” means “a lot.”

Anyway, after we turned a bunch of the apps off, her score jumped to over 90 percent of the published score. That made her happy, but she wanted to know why the apps would affect her score because “it’s the same phone.” I asked if she would expect her car to get the same gas mileage if it were carrying a bunch of cinderblocks. “Of course not,” she said, quickly understanding the metaphor. Asking your phone to do other things during the test is going to put a drag on the system.

It’s important to remember that benchmarking can be a pretty specialized area. As the popularity of the XPRTs grows, they are being run more often by bright, well educated people who don’t understand benchmarking basics. For this reason, we’re putting a lot of effort into automating best practices where we can, and guiding people when necessary.

Have the XPRTs ever confused you? If so, e-mail us at benchmarkxprtsupport@principledtechnologies.com. We can use your input.

Eric

Comment on this post in the forums

Posted in WebXPRT |

Diversity

By Eric Hale

on November 7, 2013

Fall is beautiful in North Carolina. The temperature is dropping. The leaves are changing color, making the hills scarlet and orange. And, of course, the stores have been decorated for Christmas since Halloween.

As we head into the biggest shopping season of the year, it’s a great time to be getting XPRT results from the hottest devices. In the last few weeks, we’ve published results from

tablets such as the Apple iPad Air, Google Nexus 7 2, and both the Microsoft Surface 2 and Microsoft Surface Pro 2
phones such as the Apple iPhone 5c, Apple iPhone 5s, and LG G2
devices you might not have expected, such as the Amazon Kindle Fire HDX, Barnes and Noble Nook HD+, and NVIDIA Shield

The diversity of devices is nice to see. The results come from PT testing, the press, and benchmark users. Note that you don’t have to be a community member to submit results. The person who submitted the MobileXPRT Nook HD+ results was not a member. If you’ve tested something interesting, send the results on!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, MobileXPRT, Performance testing on tablets, Touch-based computing, Touch-based devices, TouchXPRT, TouchXPRT results |

Broadening our appeal

By Eric Hale

on October 31, 2013

As I mentioned last week, we’ve asked the PT design team to help improve the XPRT benchmarks. I’m learning a lot working with them. As someone who’s been involved with benchmarking a long time, it can be a shock to realize that there are people who think “ms” is a magazine, “geomean” has something to do with the environment, and “UX” sounds like it would be a great name for a band. But the fact is that most consumers don’t need to know any of these terms, so our benchmarks shouldn’t rely on them, either.

This collaboration already paying off. The PT design team rewrote the MobileXPRT FAQ, making it much more extensive, accessible, and fun to read. We think the new FAQ is greatly improved, and it’s certainly more informative. We’ll be upgrading the FAQs for the other benchmarks in the near future.

Our efforts are going far beyond FAQs. Data presentation, graphics, the basic UI design philosophy—everything is on the table. Let us know what you think by emailing benchmarkxprtsupport@principledtechnolgies.com or by posting on the forums.

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Let us know your thoughts, MobileXPRT |

The Microsoft Surface 2

By Eric Hale

on October 24, 2013

As soon as the Microsoft Surface 2 became available, we got one and have been putting it through its paces. Of course, we ran WebXPRT and TouchXPRT. The results are on the TouchXPRT and WebXPRT sites, but I’ll repeat them here along with the results for its predecessor, the Microsoft Surface RT.

	TouchXPRT	WebXPRT
Surface RT	98	167
Surface 2	284	324

TouchXPRT shows the Surface 2 to be almost three times faster than the Microsoft Surface RT, while WebXPRT shows it to be almost twice as fast.

Why the difference? The most obvious explanation is that WebXPRT depends on the browser and its implementations of JavaScript and HTML5. TouchXPRT relies less on additional software and seems to take better advantage of the underlying hardware.

While we have yet to test the Intel Core i5-based Microsoft Surface Pro 2 ourselves, others have been doing so. Interestingly, Anandtech’s review of the Surface Pro 2 included WebXPRT results from both Chrome and IE. The Chrome result was over 30 percent higher than the IE result: 1,260 vs. 960. Unfortunately, Google has not made Chrome available for the ARM-based Surface 2, so we were not able to make that comparison.

As always, please let us know any results you get on any new hardware so we can get as many results as possible in our result databases. There are lots of new products coming out in the next few weeks and we’d love your help in getting results for as many of them as possible. Thanks!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Performance testing on tablets, Touch-based computing, Touch-based devices, TouchXPRT, TouchXPRT results, WebXPRT, Windows 8.1 |

Author Archives: Eric Hale

Sounds easy, but…

TouchXPRT 2014

It’s the same phone

Diversity

Broadening our appeal

The Microsoft Surface 2

Check out the other XPRTs: