Category: Touch-based benchmarking

TouchXPRT Web test update

on November 29, 2012

On October 22, we released TouchXPRT CP1 to the community. We took the unprecedented step of releasing CP1 without any restrictions on publishing results, and since then reviews of the Microsoft Slate and the Sony Duo 11 Convertible Laptop have used TouchXPRT.

The five scenarios in CP1 focus on media manipulation. While this is an important activity on touch devices, we know this is not all people do.

Next week, we plan to release Web-based scenarios. They use HTML 5 for a variety of activities. Unlike the original scenarios in TouchXPRT CP1, there will be nothing to download. You simply browse to a URL and run the tests online. There’s nothing to set up, just browse and run.

That means that there is nothing preventing you from running these tests on pretty much any system-browser combination that supports HTML5, not just on touch-based, Windows 8 devices like the rest of TouchXPRT. That started us wondering whether these Web-based activities should be thought of as a different benchmark entirely. When these tests are available, please try them out and let us know what you think. Do you think they are worthwhile for a broader range of devices? Do you think their scenario-based emphasis is a good alternative to existing lower-level Web-based tests?

Please keep in mind that it’s not too late to give feedback on TouchXPRT CP1. Let us know how you like the scenarios on CP1 as well as what other activities you would like to see.

-Bill Catchings

Comment on this post in the forums

Posted in Benchmarking, HTML5, Performance testing on tablets, Touch-based benchmarking, Touch-based devices, TouchXPRT, Web-based testing |

Touch device first impressions

By Bill Catchings

on November 1, 2012

One of the nice benefits of working on TouchXPRT has been the opportunity to play with the latest touch devices. The latest two are Windows 8/RT tablets, the ASUS Vivo Tab RT and the Microsoft Surface. I’ve been using the Vivo Tab without a keyboard and relying solely on touch while on the Surface, I’ve been using the cool keyboard/screen cover as well as touch. Overall, I’ve found the devices and Windows 8 very interesting, but still rough around the edges. One example of interesting on the interface formerly known as Metro (IFKaM) is the elongated landscape screens. They take a bit of getting used to, but may prove to be a good choice. On the rough-around-the-edges front, the first thing I currently do on any new device is to install Dropbox so I can get to some files to play around with. Dropbox is not yet available, though it certainly will be—hopefully soon!

IFKaM has a lot to it. On the one hand, it does not seem to me as intuitive as IOS. On the other hand, I have found that if I look hard enough, I can find a way to do something I’m used to doing in Windows. For example, even though I couldn’t use Dropbox, it dawned on me that I could just mount my file server and get to some files that way. As I spend more time with these devices, I’ll have to see if the tradeoff of a longer learning curve pays off in terms of more power and flexibility.

While I ponder that, I want to put in a mention about TouchXPRT 2013 CP1. The source is now available to members. Let us know if you see areas to improve TouchXPRT or the code itself as we work the final version over the next few months. Thanks!

Bill

Comment on this post in the forums

Posted in Touch-based benchmarking, Touch-based computing, Touch-based devices, TouchXPRT |

TouchXPRT in the fast lane

By Bill Catchings

on October 25, 2012

I titled last week’s blog “Putting the TouchXPRT pedal to the metal.” The metaphor still applies. On Monday, we released TouchXPRT 2013 Community Preview 1 (CP1). Members can download it here.

CP1 contains five scenarios based on our research and community feedback. The scenarios are Beautify Photo Album, Prepare Photos for Sharing, Convert Videos for Sharing, Export Podcast to MP3, and Create Slideshow from Photos.

Each scenario gives two types of results. There’s a rate, which allows for simple “bigger is better” comparisons. CP1 also gives the elapsed time for each scenario, which is easier to grasp intuitively. Each approach has its advantages. We’d like to get your feedback on whether you’d like us to pick one of those metrics for the final version of TouchXPRT 2013 or whether it makes more sense to include both. You’ll find a fuller description of the scenarios and the results in the TouchXPRT 2013 Community Preview 1 Design overview.

While you’re looking at CP1, we’re getting the source ready to release. To check out the source, you’ll need a system running Windows 8, with Visual Studio 2012 installed. We hope to release it on Friday. Keep your eye the TouchXPRT forums for more details.

Post your feedback to the TouchXPRT forum, or e-mail it to TouchXPRTSupport@principledtechnologies.com. Do you want more scenarios? Different metrics? A new UI feature? Let us know! Make TouchXPRT the benchmark you want it to be.

As I explained last week, we released CP1 without any restrictions on publishing results. It seems that AnandTech was the first to take advantage of that. Read AnandTech’s Microsoft Surface Review to see TouchXPRT in action.

We are hoping that other folks take advantage of CP1’s capability to act as a cross-platform benchmark on the new class of Windows 8 devices. Come join us in the fast lane!

Bill

Comment on this post in the forums

Posted in Benchmark metrics, Collaborative benchmark development, Touch-based benchmarking, Touch-based devices, TouchXPRT development process, TouchXPRT release cycle, TouchXPRT results |

Tablets everywhere

By Bill Catchings

on June 29, 2012

Everyone wants in on the tablet market. This month, two software vendors have announced hardware tablets—Microsoft’s Surface and Google’s Nexus 7. Both vendors in the past relied on OEMs to create tablets using their software (Windows and Android). Both have met with limited success doing so.

Now, both are trying Apple’s strategy of controlling the hardware as well the software. Unlike Apple, however, Microsoft and Google still need to work with their OEM partners. I’m looking forward to watching that delicate dance!

I’m looking forward more, however, to being able to actually play with both of those products. I’m also looking forward to using TouchXPRT on such products. We have not given you an update in a while on TouchXPRT, but rest assured that we are hard at work on it. Once we have HDXPRT 2012 ready to go, we will give you more details on where we are with TouchXPRT and its current schedule. The touch and tablet market are heating up and we plan to be there for it. As we have indicated before, we will support Windows 8 Metro in the first version, but we see a real need for TouchXPRT to work on multiple platforms. So much to do!

Please note that today is the end of the beta test period. We appreciate the results, bugs, and suggestions you have sent so far. Feel free, however, to continue to send us any feedback or issues you find even after the official beta period is complete. After today, we can’t guarantee to be able to address them, but we will try.

Bill

Comment on this post in the forums

Posted in Performance testing on tablets, Touch-based benchmarking, Touch-based computing, Touch-based devices, TouchXPRT |

Our new baby has a name!

By Bill Catchings

on February 24, 2012

At the beginning of the year, at CES, we announced that we would start working on a touch-based benchmark that would initially run on Windows 8 Metro. We have been hard at work learning about Metro and creating the benchmark itself.

In parallel, we’ve been working on a name for the benchmark. What we settled on was Touch eXperience & Performance Ratings Tool, or TouchXPRT for short. We’re updating the Web pages with the new name and getting the domain properly set up. In the meantime, check out the logo:

Let us know what you think about the name and the logo. We are happy with both!

I’ve been reading that the Windows 8 beta should be available soon and we hope to have an alpha TouchXPRT available within a few weeks of the beta. We will need your help to critique, debug, and expand TouchXPRT from there. Hang onto your hats, these are exciting times in Benchmark Land!

Bill

Comment on this post in the forums

Posted in Touch-based benchmarking, TouchXPRT |

Web benchmarking challenges

By Bill Catchings

on January 27, 2012

I think that an important part of any touch benchmark will be a Web component. After all, the always (or almost always) connected nature ofthese devices is a critical part of their identities. I think such a Web benchmark needs to include a measurement of page load speed (how long it takes to download and render a page).

Creating such a test seems straightforward. Pick a set of sites, such as the five or ten most popular, and then time how long the home page of each takes to load. The problem, however, is that those pages are constantly changing. Every few months, most popular sites do a major redesign. That would obviously affect the results for a test and make it difficult to compare the results of a current test to one from a few months back. It is even more of a problem that the page will be different for one user than another as sites typically know things like where you are and what your computer is and adjust things to match those characteristics. And, the ads and the content of the site are constantly changing and updating. Even hitting Refresh on a page can give you different page.

Given all of those problems, how is it possible to test page loads? One way is to create pages that are similar those of leading Web sites in terms of things like size, amount of graphics, and dynamic elements. This allows the tests to be consistent over time and from different devices and locations. (Or, at least, as consistent as the variability of the Internet from moment to moment allows.) The problem with this approach, however, is that the pages will age out as Web sites update themselves and they will not be the real sites.

Such are the tradeoffs in benchmarking. The key is how to balance real science with real world considerations. What do you think? Which approach is the better balance of real science and real world?

Bill

Comment on this post in the forums

Posted in Touch-based benchmarking |

Category: Touch-based benchmarking

TouchXPRT Web test update

Touch device first impressions

TouchXPRT in the fast lane

Tablets everywhere

Our new baby has a name!

Web benchmarking challenges

Check out the other XPRTs: