Category: Application-based benchmarks

A new HDXPRT 2014 build is available

on February 9, 2017

Last fall, we identified a way to run HDXPRT 2014, originally developed for Windows 8, on Windows 10. The method involved overwriting the HDXPRT CPU-Z files with newer versions and performing a few additional pre-test configuration steps. You can read more details about those steps here.

Today, we’re releasing a new build of HDXPRT 2014 (v1.2) that eliminates the need to overwrite the CPU-Z files. The new build is available for download at HDXPRT.com. Please note that the app package is 5.08 GB, so allow time and space for the download process.

We also updated the HDXPRT 2014 User Manual to reflect changes in pre-test system configuration and to include the settings we recommend for newer builds of Windows 10.

The changes in the new build do not affect results, so v1.2 scores are comparable to v1.1 scores on the same system.

The new build ran well during testing in our labs, but issues could emerge as Microsoft releases new Windows updates. If you have any questions about HDXPRT or encounter any issues during testing, we encourage you to let us know.

We look forward to seeing your test results!

Justin

Posted in Application-based benchmarks, Collaborative benchmark development, HDXPRT, HDXPRT 2014, Performance benchmarking, Windows 10 |

HDXPRT’s future

By Bill Catchings

on November 3, 2016

While industry pundits have written many words about the death of the PC, Windows PCs are going through a renaissance. No longer do you just choose between a desktop or a laptop in beige or black. There has been an explosion of choices.

Whether you want a super-thin notebook, a tablet, or a two-in-one device, the market has something to offer. Desktop systems can be small devices on your desk, all-in-ones with the PC built into the monitor, or old-style boxes that sit on the floor. You can go with something inexpensive that will be sufficient for many tasks or invest in a super-powerful PC capable of driving today’s latest VR devices. Or you can get a new Microsoft Surface Studio, an example of the new types of devices entering the PC scene.

The current proliferation of PC choices means that tools that help buyers understand the performance differences between systems are more important than they have been in years. Because HDXPRT is one such tool, we expect demand for it to increase.

We have many tasks ahead of us as we prepare for this increased demand. The first is to release a version of HDXPRT 2014 that doesn’t require a patch. We are working on that and should have something ready later this month.

For the other tasks, we need your input. We believe we need to update HDXPRT to reflect the world of high-definition content. It’s tempting to simply change the name to UHDXPRT, but this was our first XPRT and I’m partial to the original name. How about you?

As far as tests, what should a 2017 version of HDXPRT include? We think 4K-related workloads are a must, but aren’t sure whether 4K playback tests are the way to go. What do you think? We need to update other content, such as photo and video resolutions, and replace outdated applications with current versions. Would a VR test would be worthwhile?

Please share your thoughts with us over the coming weeks as we put together a plan for the next version of HDXPRT!

Bill

Posted in Application-based benchmarks, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, HDXPRT, HDXPRT development process, Performance benchmarking, What makes a good benchmark?, Windows 10 |

Tracking device evolution with WebXPRT ’15, part 2

By Justin Greene

on October 27, 2016

Last week, we used the Apple iPhone as a test case to show how hardware advances are often reflected in benchmark scores over time. When we compared WebXPRT 2015 scores for various iPhone models, we saw a clear trend of progressively higher scores as we moved from phones with an A7 chip to phones with A8, A9, and A10 Fusion chips. Performance increases over time are not surprising, but WebXPRT ’15 scores also showed us that upgrading from an iPhone 6 to an iPhone 6s is likely to have a much greater impact on web-browsing performance than upgrading from an iPhone 6s to an iPhone 7.

This week, we’re revisiting our iPhone test case to see how software updates can boost device performance without any changes in hardware. The original WebXPRT ’15 tests for the iPhone 5s ran on iOS 8.3, and the original tests for the iPhone 6s, 6s Plus, and SE ran on variants of iOS 9. We updated each phone to iOS 10.0.2 and ran several iterations of WebXPRT ’15.

Upgrading from iOS 8.3 to iOS 10 on the iPhone 5s caused a 17% increase in web-browsing performance, as measured by WebXPRT. Upgrading from iOS 9 to iOS 10 on the iPhone 6s, 6s Plus, and SE produced web-browsing performance gains of 2.6%, 3.6%, and 3.1%, respectively.

The chart below shows the WebXPRT ’15 scores for a range of iPhones, with each iPhone’s iOS version upgrade noted in parentheses. The dark blue columns on the left represent the original scores, and the light blue columns on the right represent the upgrade scores.

As with our hardware comparison last week, these scores are the median of a range of scores for each device in our database. These scores come both from our own testing and from device reviews from popular tech media outlets.

These results reinforce a message that we repeat often, that many factors other than hardware influence performance. Designing benchmarks that deliver relevant and reliable scores requires taking all factors into account.

What insights have you gained recently from WebXPRT ’15 testing? Let us know!

Justin

Posted in Apple, Application-based benchmarks, Benchmark metrics, Benchmarking, Browser-based benchmarks, iOS, Performance benchmarking, Uncategorized, WebXPRT |

Rebalancing our portfolio

By Bill Catchings

on September 29, 2016

We’ve written recently about the many new ways people are using their devices, the growing breadth of types of devices, and how application environments also are changing. We’ve been thinking a lot about the ways benchmarks need to adapt and what new tests we should be developing.

As part of this process, we’re reviewing the XPRT portfolio. An example we wrote about recently was Google’s statement that they are bringing Android apps to Chrome OS and moving away from Chrome apps. Assuming the plan comes to fruition, it has big implications for CrXPRT, and possibly for WebXPRT as well. Another example is that once upon a time, HDXPRT included video playback tests. The increasing importance of 4K video might mean we should bring them back.

As always, we’re interested in your thoughts. Which tests do you see as the most useful going forward? Which ones do you think might be past their prime? What new areas do you like to see us start to address? Let us know!

Over the coming weeks, we’ll share our conclusions based on these market forces and your feedback. We’re excited about the possibilities and hope you are as well.

Bill

Posted in Android, Application-based benchmarks, BenchmarkXPRT development community, Chrome OS, Collaborative benchmark development, CrXPRT, Future of performance evaluation, HDXPRT, WebXPRT, What makes a good benchmark? |

A Chrome-plated example

By Eric Hale

on September 1, 2016

A couple of weeks ago, we talked about how benchmarks have to evolve to keep up with the changing ways people use their devices. One area where we are expecting a lot of change in the next few months is Chromebooks.

These web-based devices have become very popular, even outselling Macs for the first time in Q1 of this year. Chromebooks run Google Apps and a variety of third-party Chrome apps that also run on Windows, Mac, and Linux systems.

Back in May, Google announced that Android apps would be coming to Chromebooks. This exciting development will bring a lot more applications to the platform. Now, Google has announced that they will be “moving away” from the Chrome apps platform and will be phasing out Chrome app support on other platforms within the next two years.

Clearly, the uses of Chromebooks are likely to change a lot in coming months. Interestingly, part of the rationale Google gives for this decision is the development of powerful new Web APIs, which will have implications for WebXPRT as well.

As we’ve said before, we’ll be watching and adapting as the applications change.

Eric

Posted in Application-based benchmarks, Benchmarking, Chrome OS, Future of performance evaluation, Google, WebXPRT |

Getting it right

By Eric Hale

on June 2, 2016

Back in April Bill announced that we are working on a cross-platform benchmark. We asked for your thoughts and comments, and you’ve been great! We really appreciate all the great ideas.

We’ve been using code from MobileXPRT and TouchXPRT as the basis for some experiments. In his post, Bill talked about the difficulty of porting applications. However, even though we have expertise in porting applications, it’s proving more difficult than we originally thought. Benchmarks are held to a higher standard than most applications. It’s not enough for the code to run reliably and efficiently, it must compare the different platforms fairly.

One thing we know for sure: getting it right is going to take a while. However, we owe it to you to make sure that the benchmark is reliable and fair on all platforms it supports. We will, of course, keep you informed as things progress.

In the meantime, keep sending your ideas!
Eric

Posted in Android, Application-based benchmarks, Benchmark metrics, Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, iOS, MobileXPRT, Performance benchmarking, TouchXPRT, WebXPRT, Windows 10 |