Search Results for: webxprt

More than the sum of its parts

on October 22, 2015

There was a recent article in Bloomberg about phone maker ZTE’s increasing market share in the US. The article singled out one phone, the ZTE Maven, which costs about $60 (US).

This phrase jumped out at me: “a processor with capabilities somewhere between the iPhone 5 and 6.” The iPhone 5S could also fit that description. The ZTE Maven uses the ARM Cortex-A53, 64-bit processor running at 1.2 GHz. The Apple iPhone 5s uses the Apple Cyclone-A7 Cortex-A7 Harvard Superscalar processor running at 1.3 GHz.

We decided to put that statement to the test. We ran WebXPRT 2015 on the ZTE Maven and its score was 47. The iPhone 5s scored 100. The Maven was not even close.

As we’ve said before, the performance of a device depends on more than the GHz of its processor. For example, the ZTE Maven uses the Snapdragon 410 SoC, which was aimed at mid-level devices. The iPhone 5s uses the Apple A7, which was intended for higher-end devices. You can find side by side specs here.

Be wary when you see unsupported performance claims. As this example shows, specs can appear comparable even when the actual performance of the devices differs considerably. A good benchmark can provide insights into performance that specs alone can’t.

Eric

Posted in Android, Benchmark metrics, Benchmarking, WebXPRT, WebXPRT 2015 |

Question we get a lot

By Eric Hale

on October 2, 2015

“How come your benchmark ranks devices differently than [insert other benchmark here]?” It’s a fair question, and the reason is that each benchmark has its own emphasis and tests different things. When you think about it, it would be unusual if all benchmarks did agree.

To illustrate the phenomenon, consider this excerpt from a recent browser shootout in VentureBeat:

While this looks very confusing, the simple explanation is that the different benchmarks are testing different things. To begin with, SunSpider, Octane, JetStream, PeaceKeeper, and Kraken all measure JavaScript performance. Oort Online measures WebGL performance. WebXPRT measures both JavaScript and HTML 5 performance. HTML5Test measures HTML5 compliance.

Even with benchmarks that test the same aspect of browser performance, the tests differ. Kraken and SunSpider both test the speed of JavaScript math, string, and graphics operations in isolation, but run different sets of tests to do so. PeaceKeeper profiles the JavaScript from sites such as YouTube and FaceBook.

WebXPRT, like the other XPRTs, uses scenarios that model the types of work people do with their devices.

It’s no surprise that the order changes depending on which aspect of the Web experience you emphasize, in much the same way that the most fuel-efficient cars might not be the ones with the best acceleration.

This is a bigger topic than we can deal with in a single blog post, and we’ll examine it more in the future.

Eric

Posted in Benchmark metrics, Benchmarking, Benchmarking computing devices, BenchmarkXPRT development community, Collaborative benchmark development, Performance benchmarking, Performance of computing devices, WebXPRT, What makes a good benchmark? |

BenchmarkXPRT in China

By Eric Hale

on September 10, 2015

Last week, we talked about some of the changes we’re making to the BenchmarkXPRT site to make it easier to use. This week, we’d like to talk a bit about improvements we’ve been making to support our users in China.

As you may remember, the first of the XPRTs to have a Chinese UI was BatteryXPRT. We’ve since released WebXPRT 2015 and MobileXPRT 2015, both of which have also have Chinese UIs. We’re also in the process of getting MobileXPRT 2015 listed in several major Chinese app stores. (MobileXPRT 2013 is currently available from Xiaomi and Zhushou 360.)

In other words, we’re always thinking of ways to enhance the XPRT experience for our users in China. To improve download speeds, we’ve long hosted WebXPRT on a mirror site in Singapore. Recently, based on feedback from our users and our own analysis, we’ve changed the way that the privacy notice is displayed on that site. The change allows you to run WebXPRT without loading any Google analytics, which means faster load times for all users.

We will continue to work to improve our localization. This is an area where we can use the help of the community. If you have translation skills and want to contribute the strings for a UI in your language, let us know.

Eric

Last week in the XPRTs

We added a new TouchXPRT result
We added a new HDXPRT result

Posted in BatteryXPRT 2014 for Android, BenchmarkXPRT, BenchmarkXPRT development community, Chinese, Collaborative benchmark development, MobileXPRT, MobileXPRT 2013 source, MobileXPRT 2015, WebXPRT 2015 |

What’s in a name?

By Eric Hale

on July 30, 2015

A couple of weeks ago, the Notebookcheck German site published a review of the Huawei P8lite. We were pleased to see they used WebXPRT 2015, and the P8 Lite got an overall score of 47. This week, AnandTech published their review of the Huawei P8lite. In their review, the P8lite got an overall score of 59!

Those scores are very different, but it was not difficult to figure out why. The P8lite comes in two versions, depending on your market. The version Notebookcheck used is based on HiSilicon’s Kirin 620, while the version AnandTech used was Qualcomm’s Snapdragon 615 SoC. It’s also worth noting that the phone Notebookcheck tested was running Android 5.0, while the phone AnandTech tested was running Android 4.4. With different hardware and different operating systems, it’s no surprise that the results were different.

One consequence of the XPRTs being used across the world is that is that it is not uncommon to see results from devices in different markets. As we’ve said before, many things can influence benchmark results, so don’t assume that two devices with the same name are identical.

Kudos to both AnandTech and Notebookcheck for their care in presenting the system information for the devices in their reviews. The AnandTech review even included a brief description of the two models of the P8lite. This type of information is essential for helping people make informed decisions.

In other news, Windows 10 launched yesterday. We’re looking forward to seeing the TouchXPRT and WebXPRT results!

Eric

Posted in Benchmark metrics, Benchmarking computing devices, WebXPRT 2015, Windows 10 |

An update on MobileXPRT 2015

By Eric Hale

on July 23, 2015

As I mentioned last week, we’ve been testing MobileXPRT 2015.

We’ve rebuilt the current MobileXPRT as a 64-bit application. This means that MobileXPRT 2015 will only run on Android 5.0 and above. For this reason, we’ll make MobileXPRT 2013 available for testing older versions of the operating system. Because the workloads haven’t changed, you’ll be able to compare results from MobileXPRT 2013 to those from MobileXPRT 2015.

Another change to MobileXPRT 2015 will be the removal of the UX tests. While these tests were useful on lower-end devices when MobileXPRT 2013 came out, they don’t distinguish modern devices. However, should you need them, these tests will continue to be available as part of MobileXPRT 2013.

There are a few other features that we’ll be talking about over the next few weeks. We’ll be releasing the MobileXPRT 2015 community preview before Android M is released. We’re looking forward to see how Android M performs!

Eric

Posted in Android, Android L, Android M, Collaborative benchmark development, Community Preview, MobileXPRT 2015 |

One now, one later

By Eric Hale

on July 9, 2015

Windows 10 has been on our mind this week.

Last week, we explained why the Notes test in WebXPRT would not complete when running in Edge on Windows 10. We’ve implemented the fix we discussed and have finished testing the updated versions of WebXPRT 2013 and WebXPRT 2015. We’ll release them by the end of the week. Results from the new versions are comparable with results from the existing versions.

In the current Windows 10 Mobile Beta, WebXPRT 2015 does not scroll correctly in portrait mode. It does scroll correctly in landscape mode, so, as a workaround, one can run it that way on the Windows 10 Mobile Beta.

Speaking of Windows 10 Mobile, we’ve talked before about TouchXPRT 2016 and how its purpose is to compare Windows 10 across different device types. However, Microsoft has said that Windows 10 Mobile won’t be available until after the release of Windows 10 on PCs. More importantly, the APIs and development tools won’t be final until July 29. Once Microsoft releases those tools, we’ll do our builds and tests and release a community preview.

That being said, TouchXPRT 2014 is the tool to use for comparing Windows 8.1 and Windows 10. By the time mobile devices running Windows 10 are available, TouchXPRT 2016 will be available.

Eric

Posted in TouchXPRT, TouchXPRT 2014, TouchXPRT 2016, TouchXPRT development process, WebXPRT, WebXPRT 2015, Windows 10, Windows 8.1 |

Search Results for: webxprt

More than the sum of its parts

Question we get a lot

BenchmarkXPRT in China

Last week in the XPRTs

What’s in a name?

An update on MobileXPRT 2015

One now, one later

Check out the other XPRTs: