Category: HDXPRT

Some good questions

on January 24, 2013

On Tuesday, we had a Webinar for the BenchmarkXPRT community. This Webinar covered the material that Bill would have given in individual presentations at CES. As such, it was an overview of the XPRT family.

The Webinar was well attended. We will be posting the slides and the recording of the Webinar online soon. However, we got some good questions, and thought we’d share our responses with you.

How will updates to TouchXPRT, and other benchmarks, affect results? We will avoid affecting results as much as possible. However, when updates do affect results, we will disclose the effect and the testing we performed to verify it.

Will we provide a way for benchmark users to talk to each other about support issues, perhaps via OpenBlog? We had envisioned the benchmark forums providing this opportunity. However, we are very happy to look into ways to make community communication easier and more effective.

Do you provide company memberships, as opposed to individual memberships? Not currently, although we will certainly look into this. We have no formal voting mechanism, as SPEC and some other organizations have. We may get there one day, but it’s not currently an issue. If your concern is about paying multiple membership fees, contact us, and we’ll work with you to avoid that.

In HDXPRT, can you select the CPU or GPU for video conversion and control the quality of the conversion? We have not investigated this. HDXPRT installs the applications using the default settings. However, because HDXPRT installs the applications in a separate step from running the test, it might be possible to manually change the benchmark settings and then run HDXPRT. We will be looking into this and reporting on it going forward.

How does the server influence WebXPRT results? We have run WebXPRT hosted on different servers in different locations, and seen little influence on the results. However, as part of preparing the WebXPRT general release, we will characterize and document the influence of the server.

Feel free to let us know what you think about these or any other topics. As I said earlier, we’ll be posting the whole Webinar online soon.

Eric

Comment on this post in the forums

Posted in Benchmarking, Benchmarks in general, CES, HDXPRT, TouchXPRT, WebXPRT |

Unchecked ambition

By Bill Catchings

on January 17, 2013

With 10 applications and five scenarios, HDXPRT 2012 is the most ambitious version of HDXPRT ever. However, as we said in the blog post of the same name, There is such a thing as too much. We heard that message from you as well: HDXPRT 2012 is too big. At 11 GB, we can’t make it available for download, and it won’t fit on a single DVD.

It also takes longer to use than many of you would like. The installation takes a couple of hours, and each iteration takes 2 to 3 hours. Because a valid run of HDXPRT 2012 comprises three iterations, getting a single result takes all day.

As we work to reduce the size of HDXPRT 2013, we’re being careful. While we need to shrink the benchmark and its running time, we want to be sure that we don’t compromise its essential value. As part of this process, we’re looking at both the selection of applications and the ways we can reengineer the scenarios.

If you have ideas about HDXPRT 2012 that you haven’t sent yet, now is a good time to do that. We’ll be sending out the HDXPRT 2013 RFC in the next couple of weeks. Once it’s out, we will look forward to hearing your comments.

I also wanted to mention that as we had to cancel CES at the last minute, we’re planning to do a Webinar next week (Tuesday, January 22) to cover the material we planned to present there. We’ll send out an email later this week with more details.

Bill

Comment on this post in the forums

Posted in BenchmarkXPRT, CES, HDXPRT, HDXPRT 2013, HDXPRT Development Community benefits, Let us know your thoughts |

The umbrella is open

By Eric Hale

on January 11, 2013

Last week, we told you that we were going to be reorganizing the XPRT benchmarks under the BenchmarkXPRT umbrella. We’ve now taken a big step forward. The BenchmarkXPRT.com Web site is live.

BenchmarkXPRT.com is now the one place to go for news about all the benchmarks. When you visit the HDXPRT site, you’ll notice that the TouchXPRT and WebXPRT links are gone. The HDXPRT.com site is now specific to HDXPRT, and so matches the TouchXPRT.com and WebXPRT.com sites.

As part of this effort, we have also created a new Facebook page and Twitter feed for BenchmarkXPRT and they are already active. However, so that no one misses any important information, we will redundantly post the news to the old HDXPRT page and tweet to the old HDXPRT account until April 1.

The next step in the process will be converting the existing HDXPRT Development Community memberships to be BenchmarkXPRT Development community memberships. The new memberships will work like the ones you have now. However, you’ll also have the ability to follow specific benchmarks. That means that any time there’s news related to a specific benchmark, you’ll be notified. In the next couple of days, we will send an email to the members explaining this in more detail and giving the specific date of the cut over.

We will also be renaming the HDXPRT forums to be BenchmarkXPRT forums, and reorganizing the forums themselves.

As we’ve said before, this new organization better fits with the reality now, and gives us flexibility for the future. Who knows how when the family might grow again?

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, HDXPRT, Let us know your thoughts, Principled Technologies |

Ending the year with a bang!

By Eric Hale

on January 2, 2013

As we promised in the blog post The newest member of the family, we made the WebXPRT 2013 community preview available this week. It has already been used in a review! The AnandTech review of the Acer Iconia W510 includes results from the WebXPRT 2013 community preview for that device and for the Microsoft Surface RT and the Apple iPad 4. The review has results from the TouchXPRT 2013 community preview for the Acer Iconia W510, Microsoft Surface RT, and the ASUS VivoTab RT as well.

Obviously, we’ve been doing some testing ourselves. Here’s a sampling of the devices on which we’ve successfully run WebXPRT:

Device	Processor	Operating system	Browser	Score	Confidence interval
HP Envy 2	1.8 GHz Intel Atom Z2760	Windows 8	Internet Explorer 10.0.92	201	+/- 6
Asus VivoTab RT	1.2 GHz Tegra 3 T30L	Windows RT	Internet Explorer 10.0.92	160	+/- 5
Kindle Fire	1.2 GHz ARM Cortex-A9	Android OS 2.3 (customized: 6.3.1_user_4107720)	Safari 5	92	+/- 2
ASUS-made Google Nexus 7	1.2 GHz Tegra 3 T30L	Android 4.2	Chrome 18	201	+/- 4
Motorola DroidX phone	1 GHz TI OMAP3630-1000	Android 4.5.621	Browser version 2.3.4	26	+/- 1
iPhone 5	1.3 GHz Apple A6	iOS 6.0.2	Safari 6	168	+/- 2
iPad mini	1GHz Apple A5	iOS 6.0.2	Safari 6	110	+/- 1
iPad 4	1.4 GHz Apple A6X	iOS 6.0.1	Safari 6	180	+/- 2

As the results above show, WebXPRT can run on a wide range of devices. We are working to get results on lots of different devices and would like your help. We’ll set up a forum thread for results that starts with these. We’ll then add additional ones we produce. Please respond in the thread with results you get.

In addition to performance results, the WebXPRT 2013 community preview also provides a report on the HTML 5 capabilities of your device. For those who want to know more about the capabilities of HTML 5, there’s more good news. The W3C community released the feature-complete spec for HTML 5 and Canvas 2D this week.

You can find an explanation of scenarios in the WebXPRT 2013 community preview, and an explanation of how it calculates its results in the WebXPRT 2013 CP1 Overview. Let us know what you think. There’s still time to help us shape the final version of both WebXPRT 2013 and TouchXPRT 2013.

Eric

Comment on this post in the forums

Posted in Benchmarking, HTML5, Let us know your thoughts, Performance testing on tablets, TouchXPRT, Web-based testing, WebXPRT |

The newest member of the family

By Eric Hale

on December 14, 2012

In his blog post TouchXPRT Web test update, Bill mentioned that we would be releasing Web-based workloads for the community to try out. Although we developed them as part of TouchXPRT, the cross-platform nature of these tests suggested to us that they should stand on their own.

After seeking community input, we have decided to make them a separate benchmark. So, we are proud to announce WebXPRT. The first community preview will be available mid-next week. WebXPRT CP1 contains four workloads: Photo Effects, Face Detect, Stocks Dashboard, and Offline Notes. Because the workloads are all HMTL5 based, they run on a wide variety of devices and operating systems—from iPad tablets to Android phones to Windows computers.

As with all community previews, we are very interested in your opinion. Tell us what you like or don’t like about the workloads. Are there other use cases you’d like to see?

Now that we have three benchmarks, the old HDXPRT-centric model of the community needs updating. Earlier this week, a message went out to the community announcing that we will be reorganizing the benchmarks under the umbrella of BenchmarkXPRT. This reorganization will touch all aspects of the community, from the Web site to Facebook, Twitter, and even the memberships themselves. We’ll be rolling out these changes over the next few weeks, and we’ll keep you informed every step of the way.

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, HTML5, Let us know your thoughts, Web-based testing, WebXPRT |

There is such a thing as too much

By Eric Hale

on December 6, 2012

There’s been a lot of excitement about TouchXPRT recently. However, we haven’t been ignoring HDXPRT. On November 9, we released a patch that lets HDXPRT support Windows 8. We’ve now integrated the patch into HDXPRT2012, so all copies of HDXPRT 2012 going forward will install on Windows 8 without the need for a separate step.

As promised, we will be releasing the source code for HDXPRT 2012. We anticipate having it available for community members by December 14.

During the comment period for HDXPRT, this message came through loud and clear: HDXPRT 2012 is too big and takes too long to run. So we are working hard to find the best way to reduce the number of applications and scenarios. While we want to make the benchmark smaller and faster, we want to make sure that HDXPRT 2013 is comprehensive enough to provide useful performance metrics for the greatest number of people.

We’re working toward having an RFC in late January that will define a leaner, meaner HDXPRT 2013, and will reflect the other comments we have as received as well. If you have thoughts about which applications and scenarios are most important to you, please let us know.

In other news, CES is coming in January, and Principled Technologies will be there! Once again, Bill is hoping to meet with as many of you in the Development Community as possible. We’ll have a suite at the Hilton and would love for you to come, kick back, and talk about HDXPRT, TouchXPRT, the future of benchmarks, or about the cool things you’ve seen at the show. (Bill loves talking about gadgets. Last year, he went into gadget overload!)

If you plan to be at CES, but are stuck working a booth or suite, let us know and Bill will try to stop by and say hi. Drop us an email at hdxrpt_CES@principledtechnologies.com and we will set up an appointment.

Finally, we’re really excited about the big changes at the Principled Technologies Web site. The new Web site gives us a lot of opportunities. Over the next few weeks, we’ll be looking at ways the Development Community can take advantage of them.

Eric

on this post in the forums

Posted in Application-based benchmarks, Benchmarking, CES, Collaborative benchmark development, HDXPRT 2013, HDXPRT Development Community benefits, Principled Technologies, What makes a good benchmark?, Windows 8 | Tagged CES |

Category: HDXPRT

Some good questions

Unchecked ambition

The umbrella is open

Ending the year with a bang!

The newest member of the family

There is such a thing as too much

Check out the other XPRTs: