Author Archives: Justin Greene

Tracking device evolution with WebXPRT ’15, part 2

on October 27, 2016

Last week, we used the Apple iPhone as a test case to show how hardware advances are often reflected in benchmark scores over time. When we compared WebXPRT 2015 scores for various iPhone models, we saw a clear trend of progressively higher scores as we moved from phones with an A7 chip to phones with A8, A9, and A10 Fusion chips. Performance increases over time are not surprising, but WebXPRT ’15 scores also showed us that upgrading from an iPhone 6 to an iPhone 6s is likely to have a much greater impact on web-browsing performance than upgrading from an iPhone 6s to an iPhone 7.

This week, we’re revisiting our iPhone test case to see how software updates can boost device performance without any changes in hardware. The original WebXPRT ’15 tests for the iPhone 5s ran on iOS 8.3, and the original tests for the iPhone 6s, 6s Plus, and SE ran on variants of iOS 9. We updated each phone to iOS 10.0.2 and ran several iterations of WebXPRT ’15.

Upgrading from iOS 8.3 to iOS 10 on the iPhone 5s caused a 17% increase in web-browsing performance, as measured by WebXPRT. Upgrading from iOS 9 to iOS 10 on the iPhone 6s, 6s Plus, and SE produced web-browsing performance gains of 2.6%, 3.6%, and 3.1%, respectively.

The chart below shows the WebXPRT ’15 scores for a range of iPhones, with each iPhone’s iOS version upgrade noted in parentheses. The dark blue columns on the left represent the original scores, and the light blue columns on the right represent the upgrade scores.

As with our hardware comparison last week, these scores are the median of a range of scores for each device in our database. These scores come both from our own testing and from device reviews from popular tech media outlets.

These results reinforce a message that we repeat often, that many factors other than hardware influence performance. Designing benchmarks that deliver relevant and reliable scores requires taking all factors into account.

What insights have you gained recently from WebXPRT ’15 testing? Let us know!

Justin

Posted in Apple, Application-based benchmarks, Benchmark metrics, Benchmarking, Browser-based benchmarks, iOS, Performance benchmarking, Uncategorized, WebXPRT |

Tracking device evolution with WebXPRT ‘15

By Justin Greene

on October 20, 2016

The XPRT Spotlight on the Apple iPhone 7 Plus gives us a great opportunity to look at the progression of WebXPRT 2015 scores for the iPhone line and see how hardware and software advances are often reflected in benchmark scores over time. This week, we’ll see how the evolution of Apple’s mobile CPU architecture has boosted web-browsing performance. In a future post, we’ll see the impact of iOS development.

As we’ve discussed in the past, multiple factors can influence benchmark results. While we’re currently using the iPhone as a test case, the same principles apply to all types of devices. We should also note that WebXPRT is an excellent gauge of expected web-browsing performance during real-world tasks, which is different than pure CPU performance in isolation.

When looking at WebXPRT ’15 scores in our database, we see that iPhone web-browsing performance has more than doubled in the last three years. In 2013, an iPhone 5s with an Apple A7 chip earned an overall WebXPRT ’15 score of 100. Today, a new iPhone 7 Plus with an A10 Fusion chip reports a score somewhere close to 210. The chart below shows the WebXPRT ’15 scores for a range of iPhones, with each iPhone’s CPU noted in parentheses.

Moving forward from the A7 chip in the iPhone 5s to the A8 chip in the iPhone 6 and the A9 chip in the iPhone 6s and SE, we see consistent score increases. The biggest jump, at over 48%, appears in the transition from the A8 to the A9 chip, implying that folks upgrading from an iPhone 6 or 6 Plus to anything newer would notice a huge difference in web performance.

In general, folks upgrading from an A9-based phone (6S, 6S Plus, or SE) to an A10-based phone (7 and 7 Plus) could expect an increase in web performance of over 6.5%.

The scores we list represent the median of a range of scores for each device in our database. These scores come from our own testing, as well as from device reviews from media outlets such as AnandTech, Notebookcheck, and Tom’s Hardware. It’s worth noting that the highest A9 score in our database (AnandTech’s iPhone SE score of 205) overlaps with the lowest A10 Fusion score (Tom’s Hardware of Germany’s iPhone 7 score of 203), so while the improvement in median scores is clear, performance will vary according to individual phones and other factors.

Soon, we’ll revisit our iPhone test case to see how software updates can boost device performance without any changes in hardware. For more details on the newest iPhones, visit the Spotlight comparison page to see how iPhone 7 and 7 Plus specs and WebXPRT scores stack up.

Justin

Posted in Apple, Benchmark metrics, Benchmarking, Performance benchmarking, WebXPRT, WebXPRT 2015 |

An exciting milestone for WebXPRT!

By Justin Greene

on October 5, 2016

If you’re familiar with the run counter on WebXPRT.com, you may have noticed that WebXPRT recently passed a pretty significant milestone. Since we released WebXPRT 2013, users running WebXPRT 2013 and 2015 have successfully completed over 100,000 runs!

We’re thrilled about WebXPRT’s ongoing popularity, and we think that it’s due to the benchmark’s unique combination of characteristics: it’s easy to run, it runs quickly and on a wide variety of platforms, and it evaluates device performance using real-world tasks. Manufacturers, developers, consumers, and media outlets in more than 358 cities, from Aberdeen to Zevenaar, and 57 countries, from Argentina to Vietnam, have used WebXPRT’s easy-to-understand results to compare how well devices handle everyday tasks. WebXPRT has definitely earned its reputation as a “go-to” benchmark.

If you haven’t run WebXPRT yet, give it a try. The test is free and runs in almost any browser.

We’re grateful for everyone who’s helped us reach this milestone. Here’s to another 100,000 runs!

Justin

Posted in BenchmarkXPRT development community, WebXPRT, WebXPRT 2013, WebXPRT 2015, What makes a good benchmark? |

XPRT Spotlight: making it happen

By Justin Greene

on September 22, 2016

This week’s XPRT Weekly Tech Spotlight features the Apple iPhone 7, one of the bigger launches in a year of relatively few big phone releases. We like to feature a wide array of devices in Spotlight, but events like the iPhone 7 launch are good opportunities for us to provide quick data for buyers who are considering taking the plunge.

As those of you who need to have the new hot device on the first day know, getting that device can be a trial. Even though we preordered our iPhone 7, last Friday I found myself standing in line at the Apple store for almost two hours. However, that was much shorter than the half-day-plus wait for those who hadn’t ordered ahead.

We also ordered an iPhone 7 Plus. We’ll feature it in Spotlight as soon as it arrives, but we don’t expect it to ship until October.

Have you waited in line for a popular device this year? We’d love to hear your story. As always, if there are any devices that you’d like to see in Spotlight, let us know!

Justin

Posted in Apple, XPRT Weekly Tech Spotlight |

An anniversary update

By Justin Greene

on July 21, 2016

The Windows 10 Anniversary Update release is scheduled for August 2, and we’ve been running the XPRTs on the Windows Insider preview builds. While we can’t publish performance data from developer builds, we’re happy to say that WebXPRT and TouchXPRT run well on the Anniversary Update.

The story for HDXPRT 2014 is more complicated. Back in May, we reported that it would not run on more recent versions of Windows. However, we’ve identified steps that enable HDXPRT to run on the current stable Windows 10 build, as well as the latest Anniversary Update preview. It’s running well, but it’s possible that testers will encounter other issues as Microsoft releases new builds.

We have included the steps below. We’re considering an update to HDXPRT 2014 that will incorporate these changes. If you have any comments or suggestions related to HDXPRT, please let us know.

Justin

Summary
In addition to the normal system configuration requirements for HDXPRT, testers must also overwrite HDXPRT’s CPU-Z files with newer versions and change the default browser from Microsoft Edge to Internet Explorer. After configuring the system for HDXPRT testing, testers may encounter errors related to administrative privileges when attempting to launch Microsoft Edge. Returning User Account Control settings to their default pre-configuration state resolves the problem.

Process
1. Install the latest version of CPU-Z.
      a. Open any browser and download the latest version of CPU-Z for Windows
          (currently CPU-Z 1.76).
      b. Install CPU-Z on the system, using the default settings and installation path.
2. Install the HDXPRT 2014 benchmark using the default installation process. Reboot the system
    after installation.
3. Copy all the files from the C:\Program Files\CPUID\CPU-Z\ directory to the C:\Program Files
    (x86)\HDXPRT\bin, and overwrite the existing CPU-Z files.
4. Change the default browser from Microsoft Edge to Internet Explorer:
      a. Open the Windows Settings app and select System/Default apps.
      b. Under Web browser, click the Edge icon, and select Internet Explorer from the list.
      c. At the Before you switch window, click Switch anyway.
      d. Close the Settings app.
5. Adjust SmartScreen and security settings:
      a. Open Internet Explorer.
      b. Go to Settings/Internet options/Security, and make the following changes for the Internet
           and Trusted Sites zones:
            i. Select Custom Level.
            ii. Disable SmartScreen Filter.
            iii. Under Launching applications and unsafe files, click Enable (not Secure).
            iv. Click OK, and click Apply. If a warning message appears, click Yes.
6. Restart the system.
7. Open HDXPRT and run the benchmark normally.

If, after installing HDXPRT, you encounter an error related to administrative permissions when trying to open Microsoft Edge, return User Account Controls to the default setting, and restart the system. The default User Account Control setting is the third notch from the bottom: “Notify me only when apps try to makes changes to my computer.”

Posted in BenchmarkXPRT development community, Collaborative benchmark development, HDXPRT, HDXPRT 2014, Performance benchmarking, TouchXPRT, Uncategorized, WebXPRT 2015, Windows 10, Windows 10 Anniversary Update |

Sleep studies

By Justin Greene

on May 12, 2016

Last week, we discussed the increasing complexity of power options in Android 6.0. Features such as Doze and App Standby have changed the way that the operating system manages app activity, and the wide array of UI skins used by many vendors ensures that the steps needed for pre-test configuration differ considerably from device to device.

Managing Android’s Doze feature is critical to getting a good BatteryXPRT score. To show how involved this process can be, we thought it might be helpful to present the steps for one device. Below my sig are the configuration steps we used for the Huawei Mate 8, which we recently featured in the XPRT Weekly Tech Spotlight. For other phones we’ve tested, the steps have been quite different. We’re working on distilling our experience for our tips and tricks document, and the updated version of the document will be available soon. If you have any useful tips, please let us know.

Justin

Whitelist BatteryXPRT (there are two ways to do this)
1) Access Battery manager from Settings/Advanced settings or from the Phone Manager app on the home screen.
2) Select Protected apps.
3) Use the toggles beside BatteryXPRT and BatteryXPRT Tests to allow them to keep running after the screen turns off.

Configure sleep settings
1) Open Settings from the home screen.
2) Select Display.
3) Select Sleep.
4) Select Never. This may reset to a default setting on its own. In our case, it reset to 10 minutes.

Configure screen lock settings
1) Open Settings from the home screen.
2) Select Advanced settings.
3) Select Security.
4) Scroll to the bottom of the list and use the toggle to turn off Screen lock. This keeps the device screen from locking after standby periods during the test.

Posted in Android M, Android N, Battery life, BatteryXPRT 2014 for Android, Benchmarking, Mobile devices, Power consumption |

Author Archives: Justin Greene

Tracking device evolution with WebXPRT ’15, part 2

Tracking device evolution with WebXPRT ‘15

An exciting milestone for WebXPRT!

XPRT Spotlight: making it happen

An anniversary update

Sleep studies

Check out the other XPRTs: