Category: Performance testing on tablets

The Microsoft Surface 2

on October 24, 2013

As soon as the Microsoft Surface 2 became available, we got one and have been putting it through its paces. Of course, we ran WebXPRT and TouchXPRT. The results are on the TouchXPRT and WebXPRT sites, but I’ll repeat them here along with the results for its predecessor, the Microsoft Surface RT.

	TouchXPRT	WebXPRT
Surface RT	98	167
Surface 2	284	324

TouchXPRT shows the Surface 2 to be almost three times faster than the Microsoft Surface RT, while WebXPRT shows it to be almost twice as fast.

Why the difference? The most obvious explanation is that WebXPRT depends on the browser and its implementations of JavaScript and HTML5. TouchXPRT relies less on additional software and seems to take better advantage of the underlying hardware.

While we have yet to test the Intel Core i5-based Microsoft Surface Pro 2 ourselves, others have been doing so. Interestingly, Anandtech’s review of the Surface Pro 2 included WebXPRT results from both Chrome and IE. The Chrome result was over 30 percent higher than the IE result: 1,260 vs. 960. Unfortunately, Google has not made Chrome available for the ARM-based Surface 2, so we were not able to make that comparison.

As always, please let us know any results you get on any new hardware so we can get as many results as possible in our result databases. There are lots of new products coming out in the next few weeks and we’d love your help in getting results for as many of them as possible. Thanks!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Performance testing on tablets, Touch-based computing, Touch-based devices, TouchXPRT, TouchXPRT results, WebXPRT, Windows 8.1 |

A happy coincidence

By Bill Catchings

on May 2, 2013

I love new gadgets and even the promise of new ones. Samsung just announced the specs for their upcoming Galaxy Tab 3. Initial reactions to the specs have been somewhat muted to say the least. Basically, they’ve looked at this as only an incremental improvement to the current model. The early rumors of a larger screen and bigger improvements, which turned out to be false, surely contributed to the disappointed reactions.

That being said, some sites claim that the performance of the Galaxy Tab 3 is much higher than the Galaxy Tab 2, particularly regarding graphics. We look forward to verifying these claims ourselves.

Coincidentally, this week we have been playing with an early version of PhoneXPRT (or whatever we end up calling it). So far, things are looking good. We ran it on several devices, including a Samsung Galaxy Tab 2. Like all the XPRT benchmarks, it uses real- world scenarios, which we think result in more useful and accurate results. We’ll talk more about the scenarios in the next few weeks.

It’s a very exciting time in the Android phone and tablet market. I can’t wait to try out subsequent versions of the new benchmark on the latest and greatest Android devices!

Bill

Comment on this post in the forums

Posted in MobileXPRT, Performance testing on tablets, PhoneXPRT, Touch-based devices |

Ending the year with a bang!

By Eric Hale

on January 2, 2013

As we promised in the blog post The newest member of the family, we made the WebXPRT 2013 community preview available this week. It has already been used in a review! The AnandTech review of the Acer Iconia W510 includes results from the WebXPRT 2013 community preview for that device and for the Microsoft Surface RT and the Apple iPad 4. The review has results from the TouchXPRT 2013 community preview for the Acer Iconia W510, Microsoft Surface RT, and the ASUS VivoTab RT as well.

Obviously, we’ve been doing some testing ourselves. Here’s a sampling of the devices on which we’ve successfully run WebXPRT:

Device	Processor	Operating system	Browser	Score	Confidence interval
HP Envy 2	1.8 GHz Intel Atom Z2760	Windows 8	Internet Explorer 10.0.92	201	+/- 6
Asus VivoTab RT	1.2 GHz Tegra 3 T30L	Windows RT	Internet Explorer 10.0.92	160	+/- 5
Kindle Fire	1.2 GHz ARM Cortex-A9	Android OS 2.3 (customized: 6.3.1_user_4107720)	Safari 5	92	+/- 2
ASUS-made Google Nexus 7	1.2 GHz Tegra 3 T30L	Android 4.2	Chrome 18	201	+/- 4
Motorola DroidX phone	1 GHz TI OMAP3630-1000	Android 4.5.621	Browser version 2.3.4	26	+/- 1
iPhone 5	1.3 GHz Apple A6	iOS 6.0.2	Safari 6	168	+/- 2
iPad mini	1GHz Apple A5	iOS 6.0.2	Safari 6	110	+/- 1
iPad 4	1.4 GHz Apple A6X	iOS 6.0.1	Safari 6	180	+/- 2

As the results above show, WebXPRT can run on a wide range of devices. We are working to get results on lots of different devices and would like your help. We’ll set up a forum thread for results that starts with these. We’ll then add additional ones we produce. Please respond in the thread with results you get.

In addition to performance results, the WebXPRT 2013 community preview also provides a report on the HTML 5 capabilities of your device. For those who want to know more about the capabilities of HTML 5, there’s more good news. The W3C community released the feature-complete spec for HTML 5 and Canvas 2D this week.

You can find an explanation of scenarios in the WebXPRT 2013 community preview, and an explanation of how it calculates its results in the WebXPRT 2013 CP1 Overview. Let us know what you think. There’s still time to help us shape the final version of both WebXPRT 2013 and TouchXPRT 2013.

Eric

Comment on this post in the forums

Posted in Benchmarking, HTML5, Let us know your thoughts, Performance testing on tablets, TouchXPRT, Web-based testing, WebXPRT |

TouchXPRT Web test update

By Bill Catchings

on November 29, 2012

On October 22, we released TouchXPRT CP1 to the community. We took the unprecedented step of releasing CP1 without any restrictions on publishing results, and since then reviews of the Microsoft Slate and the Sony Duo 11 Convertible Laptop have used TouchXPRT.

The five scenarios in CP1 focus on media manipulation. While this is an important activity on touch devices, we know this is not all people do.

Next week, we plan to release Web-based scenarios. They use HTML 5 for a variety of activities. Unlike the original scenarios in TouchXPRT CP1, there will be nothing to download. You simply browse to a URL and run the tests online. There’s nothing to set up, just browse and run.

That means that there is nothing preventing you from running these tests on pretty much any system-browser combination that supports HTML5, not just on touch-based, Windows 8 devices like the rest of TouchXPRT. That started us wondering whether these Web-based activities should be thought of as a different benchmark entirely. When these tests are available, please try them out and let us know what you think. Do you think they are worthwhile for a broader range of devices? Do you think their scenario-based emphasis is a good alternative to existing lower-level Web-based tests?

Please keep in mind that it’s not too late to give feedback on TouchXPRT CP1. Let us know how you like the scenarios on CP1 as well as what other activities you would like to see.

-Bill Catchings

Comment on this post in the forums

Posted in Benchmarking, HTML5, Performance testing on tablets, Touch-based benchmarking, Touch-based devices, TouchXPRT, Web-based testing |

Putting the TouchXPRT pedal to the metal

By Bill Catchings

on October 19, 2012

Since we announced TouchXPRT early this year, we’ve been following a typical benchmark development path. We started with the most important question—“What are people likely to do with a touch-based Windows 8 device?”—and built from there. We looked at what people are doing now with iOS- and Android-based devices. We worked with early Windows 8 units. We studied app stores. We spoke with members of the development community. And so on. When we were done studying, we moved to coding.

We’re making great progress, but something has been nagging at us: When Windows 8 tablets and other devices ship next week, there just won’t be much in the way of tools for measuring their performance when running Windows 8 apps. Sure, you may be able use standard benchmarks to assess the performance of typical desktop applications, but that won’t tell you how the devices will perform with tablet apps.

So, we’ve decided to put the pedal to the metal and provide everyone in our development community with a special treat. Sometime next week, before Windows 8 ships, we plan to release a sneak preview of TouchXPRT, the TouchXPRT 2013 Community Preview 1 (CP1).

CP1, as its name makes clear, is not the final TouchXPRT release. It is, though, a useful tool for beginning to measure Windows 8 device performance. It is also a great way for everyone in the community to see the current state of our thinking and to provide us feedback—rather than read a design spec, you can actually run this version of the tool and see what you think! (If you would like to read the informal design spec, check out http://www.hdxprt.com/forum/touchxprt2013cp1.php .)

To make the tool easier to evaluate and more useful to all of us, we’re also taking two more unusual steps:

1. We’re not putting any publication restrictions on this preview release. Test at will, and publish your findings.

2. We’re releasing the source code to all community members. If you’re curious about not just what we’re doing but how we’re doing it, you can find out.

We hope these steps will speed acceptance of TouchXPRT 2013 and foster more and faster feedback. Releasing a preview version is more work, because we have to do much of the work of a software release and on less-than-final code, but we believe the value to our community justifies the effort.

Next week, when we release CP1, I’ll go over more details, the known limitations, and how you can get us your feedback—feedback we very much want.

Between now and then, we’ll be readying CP1 for your use.

Bill

Comment on this post in the forums

Posted in Performance testing on tablets, Touch-based computing, TouchXPRT, TouchXPRT development process, TouchXPRT results |

Tablets everywhere

By Bill Catchings

on June 29, 2012

Everyone wants in on the tablet market. This month, two software vendors have announced hardware tablets—Microsoft’s Surface and Google’s Nexus 7. Both vendors in the past relied on OEMs to create tablets using their software (Windows and Android). Both have met with limited success doing so.

Now, both are trying Apple’s strategy of controlling the hardware as well the software. Unlike Apple, however, Microsoft and Google still need to work with their OEM partners. I’m looking forward to watching that delicate dance!

I’m looking forward more, however, to being able to actually play with both of those products. I’m also looking forward to using TouchXPRT on such products. We have not given you an update in a while on TouchXPRT, but rest assured that we are hard at work on it. Once we have HDXPRT 2012 ready to go, we will give you more details on where we are with TouchXPRT and its current schedule. The touch and tablet market are heating up and we plan to be there for it. As we have indicated before, we will support Windows 8 Metro in the first version, but we see a real need for TouchXPRT to work on multiple platforms. So much to do!

Please note that today is the end of the beta test period. We appreciate the results, bugs, and suggestions you have sent so far. Feel free, however, to continue to send us any feedback or issues you find even after the official beta period is complete. After today, we can’t guarantee to be able to address them, but we will try.

Bill

Comment on this post in the forums

Posted in Performance testing on tablets, Touch-based benchmarking, Touch-based computing, Touch-based devices, TouchXPRT |

Category: Performance testing on tablets

The Microsoft Surface 2

A happy coincidence

Ending the year with a bang!

TouchXPRT Web test update

Putting the TouchXPRT pedal to the metal

Tablets everywhere

Check out the other XPRTs: