Category: WebXPRT

Tracking device evolution with WebXPRT ’15, part 2

on October 27, 2016

Last week, we used the Apple iPhone as a test case to show how hardware advances are often reflected in benchmark scores over time. When we compared WebXPRT 2015 scores for various iPhone models, we saw a clear trend of progressively higher scores as we moved from phones with an A7 chip to phones with A8, A9, and A10 Fusion chips. Performance increases over time are not surprising, but WebXPRT ’15 scores also showed us that upgrading from an iPhone 6 to an iPhone 6s is likely to have a much greater impact on web-browsing performance than upgrading from an iPhone 6s to an iPhone 7.

This week, we’re revisiting our iPhone test case to see how software updates can boost device performance without any changes in hardware. The original WebXPRT ’15 tests for the iPhone 5s ran on iOS 8.3, and the original tests for the iPhone 6s, 6s Plus, and SE ran on variants of iOS 9. We updated each phone to iOS 10.0.2 and ran several iterations of WebXPRT ’15.

Upgrading from iOS 8.3 to iOS 10 on the iPhone 5s caused a 17% increase in web-browsing performance, as measured by WebXPRT. Upgrading from iOS 9 to iOS 10 on the iPhone 6s, 6s Plus, and SE produced web-browsing performance gains of 2.6%, 3.6%, and 3.1%, respectively.

The chart below shows the WebXPRT ’15 scores for a range of iPhones, with each iPhone’s iOS version upgrade noted in parentheses. The dark blue columns on the left represent the original scores, and the light blue columns on the right represent the upgrade scores.

As with our hardware comparison last week, these scores are the median of a range of scores for each device in our database. These scores come both from our own testing and from device reviews from popular tech media outlets.

These results reinforce a message that we repeat often, that many factors other than hardware influence performance. Designing benchmarks that deliver relevant and reliable scores requires taking all factors into account.

What insights have you gained recently from WebXPRT ’15 testing? Let us know!

Justin

Posted in Apple, Application-based benchmarks, Benchmark metrics, Benchmarking, Browser-based benchmarks, iOS, Performance benchmarking, Uncategorized, WebXPRT |

Tracking device evolution with WebXPRT ‘15

By Justin Greene

on October 20, 2016

The XPRT Spotlight on the Apple iPhone 7 Plus gives us a great opportunity to look at the progression of WebXPRT 2015 scores for the iPhone line and see how hardware and software advances are often reflected in benchmark scores over time. This week, we’ll see how the evolution of Apple’s mobile CPU architecture has boosted web-browsing performance. In a future post, we’ll see the impact of iOS development.

As we’ve discussed in the past, multiple factors can influence benchmark results. While we’re currently using the iPhone as a test case, the same principles apply to all types of devices. We should also note that WebXPRT is an excellent gauge of expected web-browsing performance during real-world tasks, which is different than pure CPU performance in isolation.

When looking at WebXPRT ’15 scores in our database, we see that iPhone web-browsing performance has more than doubled in the last three years. In 2013, an iPhone 5s with an Apple A7 chip earned an overall WebXPRT ’15 score of 100. Today, a new iPhone 7 Plus with an A10 Fusion chip reports a score somewhere close to 210. The chart below shows the WebXPRT ’15 scores for a range of iPhones, with each iPhone’s CPU noted in parentheses.

Moving forward from the A7 chip in the iPhone 5s to the A8 chip in the iPhone 6 and the A9 chip in the iPhone 6s and SE, we see consistent score increases. The biggest jump, at over 48%, appears in the transition from the A8 to the A9 chip, implying that folks upgrading from an iPhone 6 or 6 Plus to anything newer would notice a huge difference in web performance.

In general, folks upgrading from an A9-based phone (6S, 6S Plus, or SE) to an A10-based phone (7 and 7 Plus) could expect an increase in web performance of over 6.5%.

The scores we list represent the median of a range of scores for each device in our database. These scores come from our own testing, as well as from device reviews from media outlets such as AnandTech, Notebookcheck, and Tom’s Hardware. It’s worth noting that the highest A9 score in our database (AnandTech’s iPhone SE score of 205) overlaps with the lowest A10 Fusion score (Tom’s Hardware of Germany’s iPhone 7 score of 203), so while the improvement in median scores is clear, performance will vary according to individual phones and other factors.

Soon, we’ll revisit our iPhone test case to see how software updates can boost device performance without any changes in hardware. For more details on the newest iPhones, visit the Spotlight comparison page to see how iPhone 7 and 7 Plus specs and WebXPRT scores stack up.

Justin

Posted in Apple, Benchmark metrics, Benchmarking, Performance benchmarking, WebXPRT, WebXPRT 2015 |

An exciting milestone for WebXPRT!

By Justin Greene

on October 5, 2016

If you’re familiar with the run counter on WebXPRT.com, you may have noticed that WebXPRT recently passed a pretty significant milestone. Since we released WebXPRT 2013, users running WebXPRT 2013 and 2015 have successfully completed over 100,000 runs!

We’re thrilled about WebXPRT’s ongoing popularity, and we think that it’s due to the benchmark’s unique combination of characteristics: it’s easy to run, it runs quickly and on a wide variety of platforms, and it evaluates device performance using real-world tasks. Manufacturers, developers, consumers, and media outlets in more than 358 cities, from Aberdeen to Zevenaar, and 57 countries, from Argentina to Vietnam, have used WebXPRT’s easy-to-understand results to compare how well devices handle everyday tasks. WebXPRT has definitely earned its reputation as a “go-to” benchmark.

If you haven’t run WebXPRT yet, give it a try. The test is free and runs in almost any browser.

We’re grateful for everyone who’s helped us reach this milestone. Here’s to another 100,000 runs!

Justin

Posted in BenchmarkXPRT development community, WebXPRT, WebXPRT 2013, WebXPRT 2015, What makes a good benchmark? |

Rebalancing our portfolio

By Bill Catchings

on September 29, 2016

We’ve written recently about the many new ways people are using their devices, the growing breadth of types of devices, and how application environments also are changing. We’ve been thinking a lot about the ways benchmarks need to adapt and what new tests we should be developing.

As part of this process, we’re reviewing the XPRT portfolio. An example we wrote about recently was Google’s statement that they are bringing Android apps to Chrome OS and moving away from Chrome apps. Assuming the plan comes to fruition, it has big implications for CrXPRT, and possibly for WebXPRT as well. Another example is that once upon a time, HDXPRT included video playback tests. The increasing importance of 4K video might mean we should bring them back.

As always, we’re interested in your thoughts. Which tests do you see as the most useful going forward? Which ones do you think might be past their prime? What new areas do you like to see us start to address? Let us know!

Over the coming weeks, we’ll share our conclusions based on these market forces and your feedback. We’re excited about the possibilities and hope you are as well.

Bill

Posted in Android, Application-based benchmarks, BenchmarkXPRT development community, Chrome OS, Collaborative benchmark development, CrXPRT, Future of performance evaluation, HDXPRT, WebXPRT, What makes a good benchmark? |

How we do it

By Eric Hale

on September 15, 2016

In the modern world, we’re awash in statistics and it’s interesting how they sometimes contradict each other. In this season, polls are an obvious example. They seldom agree and sometimes, as in the case of Brexit, they can be spectacularly wrong.

The real world is complicated and there are many ways to look at the data. An approach can be valid for certain cases, but less so for others. This is why it’s so important to explain your methods and calculations.

We’ve talked before about the statistics we use in the XPRTs. There are a set of white papers that describe how the tests work and how we perform the calculations. If you’re interested, a great place to start is the WebXPRT 2013 results calculation and confidence interval white paper, which describes the statistics in detail and links to a spreadsheet with a detailed example.

Any methodology can be improved, so if you have any ideas, let us know!

Eric

Posted in WebXPRT, WebXPRT 2013, WebXPRT 2013 results, White papers |

A Chrome-plated example

By Eric Hale

on September 1, 2016

A couple of weeks ago, we talked about how benchmarks have to evolve to keep up with the changing ways people use their devices. One area where we are expecting a lot of change in the next few months is Chromebooks.

These web-based devices have become very popular, even outselling Macs for the first time in Q1 of this year. Chromebooks run Google Apps and a variety of third-party Chrome apps that also run on Windows, Mac, and Linux systems.

Back in May, Google announced that Android apps would be coming to Chromebooks. This exciting development will bring a lot more applications to the platform. Now, Google has announced that they will be “moving away” from the Chrome apps platform and will be phasing out Chrome app support on other platforms within the next two years.

Clearly, the uses of Chromebooks are likely to change a lot in coming months. Interestingly, part of the rationale Google gives for this decision is the development of powerful new Web APIs, which will have implications for WebXPRT as well.

As we’ve said before, we’ll be watching and adapting as the applications change.

Eric

Posted in Application-based benchmarks, Benchmarking, Chrome OS, Future of performance evaluation, Google, WebXPRT |

Category: WebXPRT

Tracking device evolution with WebXPRT ’15, part 2

Tracking device evolution with WebXPRT ‘15

An exciting milestone for WebXPRT!

Rebalancing our portfolio

How we do it

A Chrome-plated example

Check out the other XPRTs: