Category: Performance benchmarking

Testing the waters

on June 19, 2014

A couple of weeks ago, we talked about some of our ideas for a new XPRT designed for Google’s Chrome OS. We’ve been working with some of these ideas and, while we’re still in the experimental stage, things look promising so far.

As we mentioned in the earlier blog, we’re trying WebXPRT as a base for the performance part of the test. So far, the performance component is working well. In addition to modified WebXPRT tests, we’re also trying some things that are not part of the WebXPRT 2013 workload.

We’ve been able to get battery life, but it’s been challenging and we haven’t found a way to avoid using Chrome’s Developer Mode. Accessing Developer Mode in Chrome can be tricky and requires different steps for each hardware manufacturer. We’re hoping to find ways to make battery life testing easier.

I’ve been vague about the tests because they’re likely to change over the next few weeks. We’re experimenting with both browser-based and Native Client-based performance tasks. As they firm up, I’ll be able to share more information.

Challenges aside, we’re excited about this new benchmark, and committed to making it as effective as possible. We’d still love feedback on a name, so feel free to contact us at BenchmarkXPRTsupport@principledtechnologies.com with your ideas.

Eric

Comment on this post in the forums

Posted in Battery life, Chrome OS, Collaborative benchmark development, Performance benchmarking, What makes a good benchmark? |

TouchXPRT 2014 arrives Friday!

By Eric Hale

on May 15, 2014

Friday is the big day for TouchXPRT 2014. It will be available for download at TouchXPRT.com and through the Windows App Store no later than 5:00 PM EDT on Friday May 16.

As we mentioned last week, in addition to the new look and more-demanding component tests, TouchXPRT has a number of new features. One feature we are especially excited about is that users will be able to automate their TouchXPRT tests, something they could not do in TouchXPRT 2013.

You can run TouchXPRT 2014 from a command prompt or using scripting mechanisms such as batch files or PowerShell. The scripting interface allows you to set several parameters, including:

Start running the test automatically on launch. You can choose to run all of the scenarios to generate an overall score, or run any single scenario for a component test score.
Change the number of iterations the test should run.
Change the delay before the test starts running after launch.
Change the output filename.

You can read the automation details and find specific commands in the Exploring TouchXPRT 2014 white paper.

Also, we’re in the last steps of preparing the HDXPRT 2014 Community Preview. It’s coming in the next couple of weeks, so stay tuned for more information.

Eric

Comment on this post in the forums

Posted in Community Preview, HDXPRT 2014, Performance benchmarking, TouchXPRT, TouchXPRT 2014, White papers, Windows 8, Windows 8.1 |

What is truth?

By Eric Hale

on April 3, 2014

Last fall, we discussed AnandTech’s report on benchmark cheating, and why open and honest benchmark development is so important. This week, benchmark optimization is back in the news, as AnandTech says that the HTC One M8 boosts its performance in whitelisted benchmarks. CNET has quoted HTC admitting that they not only boosted performance, but promoted the boost as a feature.

However, HTC has gone a step further, giving users the option to manually set the phone to high-performance mode. Some of us at PT have been involved in developing benchmarks for over 20 years, and it’s always been true that one person’s cheat can be another person’s valid optimization. Whatever their motivation, HTC’s position – that some people will choose higher performance over longer battery life – is not necessarily wrong.

BatteryXPRT recognizes that there’s a tradeoff between performance and battery life, and that you shouldn’t penalize a fast system the same way you would a system that simply has poor battery life. That’s why it reports a performance score along with the estimated battery life.

Do you have thoughts on optimizations, cheating, or ways to make the benchmarks better? Please drop us a line at BenchmarkXPRTSupport@principledtechnologies.com.

Eric

Comment on this post in the forums

Posted in BatteryXPRT 2014 for Android, Benchmarking, Collaborative benchmark development, Performance benchmarking, What makes a good benchmark? |

Golden tickets

By Bill Catchings

on January 9, 2014

We (Bill and Mark) are on our way home from CES. There were lots of cool things to see, from electric cars to health and fitness wearables to all manner of mobile devices. And, more, a whole lot more.

We enjoyed seeing many of those products, but that was not our primary mission at the show. Our main goal was to spread the word about the XPRT benchmarks. We did that by visiting multiple mobile-device members and giving out to many of them a very special golden ticket. Yes, we’re talking a physical, Willy Wonka-style golden ticket. The two-sided tickets look really cool.

One side invites folks to be heard by joining the BenchmarkXPRT community. The other offers them the opportunity to have PT test devices for free with all the applicable XPRT benchmarks. All a vendor has to do to get this free testing is send the device to PT. We hope to get many devices in-house and to provide a great many results on our Web sites.

We wore the new BenchmarkXPRT shirts as we walked the floor.

We will soon be sending one shirt—and one golden ticket—to each member of the community. Please make sure we have your latest mailing address so we can ship those to you.

-Bill & Mark

Comment on this post in the forums

Posted in BenchmarkXPRT development community, CES, Future of performance evaluation, Performance benchmarking, Principled Technologies |

Looking for a winner

By Eric Hale

on October 11, 2012

This week, PT published its first two public reports using HDXPRT 2012: Performance comparison: Dell Latitude E5430 vs. HP ProBook 4440s and Performance comparison: Dell Latitude E5430 vs. Lenovo ThinkPad L430. You should check them out.

Of course, you can find the HDXPRT results from these reports in the HDXPRT 2012 results database along with results from the characterization study we did last month. The results database is a repository of HDXPRT results you can use to compare system performance. The database includes full disclosure information and lets you sort by a number of criteria, including any HDXPRT score, the processor, amount of RAM, graphic card, and so on.

Looking at the results in the database got me wondering who has the mightiest machine out there. The current winner is a custom-built system with an Intel Core i7 3770 and 8 GB of RAM. It has an HDXPRT 2012 Create HD score of 248.

Records are meant to be broken, and I know someone out there can grind that score to dust. So, we’re going to have a contest. The first person to submit a set of HDXPRT results with a score above 248 will win at least bragging rights and maybe a prize if we can find something suitable around our offices.

You’ll find instructions for submitting results at Submit your HDXPRT 2012 results.

I can’t wait to see your results!

Eric

Comment on this post in the forums

Posted in Application-based benchmarks, Benchmarking, HDXPRT 2012, HDXPRT 2012 results, Performance benchmarking |

Benchmarking a benchmark

By Bill Catchings

on July 13, 2011

One of the challenges of any benchmark is understanding its characteristics. The goal of a benchmark is to measure performance under a defined set of circumstances. For system-level, application-oriented benchmarks, it isn’t always obvious how individual components in the system influence the overall score. For instance, how does doubling the amount of memory affect the benchmark score? The best way to understand the characteristics of a benchmark is to run a series of carefully controlled experiments that change one variable at a time. To test the benchmark’s behavior with increased memory, you would take a system and run the benchmark with different amounts of RAM. Changing the processor, graphics subsystem, or hard disk lets you see the influence of those components. Some components, like memory, can change in both their amount and speed.

The full matrix of system components to test can quickly grow very large. While the goal is to change only one component at a time, this is not always possible. For example, you can’t change the processor from an Intel to an AMD without also changing the motherboard.

We are in the process of putting HDXPRT 2011 through a series of such tests. HDXPRT 2011 is a system-level, application-oriented benchmark for measuring the performance of PCs on consumer-oriented HD media scenarios. We want to understand, and share with you, how different components influence HDXPRT scores. We expect to release a report on our findings next week. It will include results detailing the effect of processor speed, amount of RAM, hard disk type, and graphics subsystem.

There is a tradeoff between the size of the matrix and how long it takes to produce the results. We’ve tried to choose the areas we felt were most important, but we’d like to hear what you consider important. So, what characteristics of HDXPRT 2011 would you like to see us test?

Bill

Comment on this post in the forums

Posted in Application-based benchmarks, Benchmark metrics, HDXPRT 2011 results, Let us know your thoughts, Performance benchmarking, What makes a good benchmark? |

Category: Performance benchmarking

Testing the waters

TouchXPRT 2014 arrives Friday!

What is truth?

Golden tickets

Looking for a winner

Benchmarking a benchmark

Check out the other XPRTs: