Category: Performance benchmarking

Looking for a bargain?

on August 14, 2014

There are many benefits to being a member of the community: the XPRT community previews, the source code for the benchmarks, the monthly newsletter, and more. To join the community, all you’ve had to do up until now is sign up and pay a one-time $20 fee. Our goal with the fee was to make sure that people who joined were serious.

Today, we’re announcing a change. We recognize that, for some companies, getting that $20 fee reimbursed can be a hassle. So, if you work for a device maker, OEM, chip manufacturer, or retailer, you’ll be able to join the community for free.

Here’s how it works: Simply fill out the form, use your company e-mail address, and click the option to be considered for a free membership. We’ll send you an email within one business day to verify the address is real and then activate your membership.

Simple, right?

Justin

Comment on this post in the forums

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Performance benchmarking, Source code |

More details to come

By Eric Hale

on July 10, 2014

As we’ve been saying the past couple of months, we’re working on a benchmark for Chrome OS. The experimentation phase is winding down, and we are starting to shape the code into a useable benchmark. The design plan will leverage existing WebXPRT tests, of course. However, we’ve gone far beyond that. The benchmark will include video playback, 3D modeling via WebGL, and even an HTML5 game. The test also uses Chrome OS’ native execution capability. The benchmark will actually use the Portable Native Client (PNaCl), as PNaCl is the recommended tool chain for native client. It also gives the benchmark the ability to run on more platforms.

As we mentioned before, we’re including a battery test as part of the new benchmark. So far, we haven’t found a way to remove the requirement to put the device in developer mode for the battery test.

Next week, we’ll publish a design document for the community to review. As always, the design document is based on the comments and suggestions we received combined with our own research and experimentation.

Eric

Comment on this post in the forums

Posted in Chrome OS, Future of performance evaluation, HTML5, Performance benchmarking, WebXPRT |

It makes a difference

By Eric Hale

on July 3, 2014

Ars Technica reported this week that they tested the developer preview of Android L and saw a whopping 36 percent improvement in battery life! Google made improving battery life a priority, and it sounds like they are succeeding. I can’t wait to test Android L with BatteryXPRT.

This is a spectacular example of how a change in software can change benchmark results, but it’s hardly unique. I’ve written before about how background activity on a phone depressed my friend’s WebXPRT scores. AnandTech used both IE 11 and Chrome 30 to test the Surface Pro 2 with a variety of benchmarks, including WebXPRT, SunSpider, Octane, Browsermark, and others. Browser choice had a noticeable impact on results – about a 40 percent difference for WebXPRT and a 76 percent difference for SunSpider!

People are generally pretty aware that changing the hardware changes performance. However, sometimes they lose track of software differences. When you compare scores, it’s not always possible to keep all the variables the same, but it’s crucial to know what the differences are.

In other BenchmarkXPRT news, we’re making some final adjustments to HDXPRT 2014, and the general release is just around the corner.

Eric

Comment on this post in the forums

Posted in Android, Battery life, HDXPRT 2014, Performance benchmarking, Performance testing on tablets, WebXPRT |

Testing the waters

By Eric Hale

on June 19, 2014

A couple of weeks ago, we talked about some of our ideas for a new XPRT designed for Google’s Chrome OS. We’ve been working with some of these ideas and, while we’re still in the experimental stage, things look promising so far.

As we mentioned in the earlier blog, we’re trying WebXPRT as a base for the performance part of the test. So far, the performance component is working well. In addition to modified WebXPRT tests, we’re also trying some things that are not part of the WebXPRT 2013 workload.

We’ve been able to get battery life, but it’s been challenging and we haven’t found a way to avoid using Chrome’s Developer Mode. Accessing Developer Mode in Chrome can be tricky and requires different steps for each hardware manufacturer. We’re hoping to find ways to make battery life testing easier.

I’ve been vague about the tests because they’re likely to change over the next few weeks. We’re experimenting with both browser-based and Native Client-based performance tasks. As they firm up, I’ll be able to share more information.

Challenges aside, we’re excited about this new benchmark, and committed to making it as effective as possible. We’d still love feedback on a name, so feel free to contact us at BenchmarkXPRTsupport@principledtechnologies.com with your ideas.

Eric

Comment on this post in the forums

Posted in Battery life, Chrome OS, Collaborative benchmark development, Performance benchmarking, What makes a good benchmark? |

TouchXPRT 2014 arrives Friday!

By Eric Hale

on May 15, 2014

Friday is the big day for TouchXPRT 2014. It will be available for download at TouchXPRT.com and through the Windows App Store no later than 5:00 PM EDT on Friday May 16.

As we mentioned last week, in addition to the new look and more-demanding component tests, TouchXPRT has a number of new features. One feature we are especially excited about is that users will be able to automate their TouchXPRT tests, something they could not do in TouchXPRT 2013.

You can run TouchXPRT 2014 from a command prompt or using scripting mechanisms such as batch files or PowerShell. The scripting interface allows you to set several parameters, including:

Start running the test automatically on launch. You can choose to run all of the scenarios to generate an overall score, or run any single scenario for a component test score.
Change the number of iterations the test should run.
Change the delay before the test starts running after launch.
Change the output filename.

You can read the automation details and find specific commands in the Exploring TouchXPRT 2014 white paper.

Also, we’re in the last steps of preparing the HDXPRT 2014 Community Preview. It’s coming in the next couple of weeks, so stay tuned for more information.

Eric

Comment on this post in the forums

Posted in Community Preview, HDXPRT 2014, Performance benchmarking, TouchXPRT, TouchXPRT 2014, White papers, Windows 8, Windows 8.1 |

What is truth?

By Eric Hale

on April 3, 2014

Last fall, we discussed AnandTech’s report on benchmark cheating, and why open and honest benchmark development is so important. This week, benchmark optimization is back in the news, as AnandTech says that the HTC One M8 boosts its performance in whitelisted benchmarks. CNET has quoted HTC admitting that they not only boosted performance, but promoted the boost as a feature.

However, HTC has gone a step further, giving users the option to manually set the phone to high-performance mode. Some of us at PT have been involved in developing benchmarks for over 20 years, and it’s always been true that one person’s cheat can be another person’s valid optimization. Whatever their motivation, HTC’s position – that some people will choose higher performance over longer battery life – is not necessarily wrong.

BatteryXPRT recognizes that there’s a tradeoff between performance and battery life, and that you shouldn’t penalize a fast system the same way you would a system that simply has poor battery life. That’s why it reports a performance score along with the estimated battery life.

Do you have thoughts on optimizations, cheating, or ways to make the benchmarks better? Please drop us a line at BenchmarkXPRTSupport@principledtechnologies.com.

Eric

Comment on this post in the forums

Posted in BatteryXPRT 2014 for Android, Benchmarking, Collaborative benchmark development, Performance benchmarking, What makes a good benchmark? |

Category: Performance benchmarking

Looking for a bargain?

More details to come

It makes a difference

Testing the waters

TouchXPRT 2014 arrives Friday!

What is truth?

Check out the other XPRTs: