Category: Benchmark metrics

Focusing the spotlight

on March 24, 2016

As you may have heard, the Samsung Galaxy S7 is the XPRT Weekly Tech Spotlight this week. As we were testing it, we noticed that our WebXPRT scores were about 8 percent lower than those reported by AnandTech.

The folks at AnandTech do a good job on their reviews, so we wanted to understand the discrepancy in scores. The S7 comes in a couple of models, so we started by verifying that our model was the same as theirs. It was.

The next step was to check their configuration against ours, and this is where we found the difference. Both phones were running the same version of Android, but the S7 AnandTech tested used Chrome 48 while the S7 we tested came preloaded with Chrome 49. In our testing, we’ve noticed that upgrading from Chrome 48 to Chrome 49 has a noticeable performance impact on certain devices. On the Samsung Galaxy S6, the scores went down about 10 percent. In all cases we’ve seen, the decrease is driven largely by the Stock Option Pricing workload.

This isn’t the first time we’ve written about browser versions affecting results. WebXPRT is a browsing benchmark, and the browser has a legitimate impact on performance. When you’re comparing results, it’s always important to look at all the factors involved.

Justin

Last week in the XPRTs

We published the XPRT Weekly Tech Spotlight on the Samsung Galaxy S7.
We added two new BatteryXPRT ’14 results.
We added one new MobileXPRT ’15 result.
We added four new WebXPRT ’15 results.

Posted in AnandTech, Android, Android M, Benchmark metrics, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Google Chrome, WebXPRT, WebXPRT 2015 |

Is it hot in here?

By Eric Hale

on March 3, 2016

One of the great meetings I had at CES was with another community member, Patrick Chang, Senior System Engineer in the Tablet Performance group at Dell. I was glad to hear that, when he tests his devices, Patrick makes frequent use of TouchXPRT.

While TouchXPRT stresses the system appropriately, Patrick’s job requires him to understand not only how well the device performs, but why it performs that way. He was wondering what we could do to help him correlate the temperature, power consumption, and performance of a device.

That information is not typically available to software and apps like the XPRTs. However, it may be possible to add some hooks that would let the XPRTs coordinate with third-party utilities and hardware that do.

As always, the input from the community guides the design of the XPRTs. So, we’d love to know how much interest the community has in having this type of information. If you have thoughts about this, or other kinds of information you’d like the XPRTs to gather, please let us know!

Eric

Last week in the XPRTs
We published the XPRT Weekly Tech Spotlight on the Apple iPad Pro.
We added one new BatteryXPRT ’14 result.
We added one new CrXPRT ’15 result.
We added one new MobileXPRT ’13 result.
We added four new WebXPRT ’15 results.

Posted in Benchmark metrics, Benchmarking, BenchmarkXPRT, CES, Collaborative benchmark development, Dell, Future of performance evaluation, Patrick Chang, Performance benchmarking, Power consumption, Temperature, TouchXPRT, Trade Shows, What makes a good benchmark? |

The XPRT Women Code-a-Thon

By Eric Hale

on January 28, 2016

As Justin explained last week, we’ve resolved the issue we found with the TouchXPRT CP. I’m happy to say that the testing went well and that we released CP3 this week.

It’s been only three weeks since we announced the XPRT Weekly Tech Spotlight, and we already have another big announcement! Principled Technologies has joined with ChickTech Seattle to host the first ever XPRT Women Code-a-Thon! In this two-day event, participants will compete to create the best new candidate workload for WebXPRT or MobileXPRT. The workloads can’t duplicate existing workloads, so we are looking forward to seeing the new ideas.

Judges will study all the workloads and award prizes to the top three: $2,500 for first place, $1,500 for second place, and $1,000 for third place. Anyone interested can register here.

PT and the BenchmarkXPRT Development Community are committed to promoting the advancement of women in STEM, but we also win by doing good. As with the NCSU senior project, the BenchmarkXPRT Development Community will get some fresh perspectives and some new experimental test tools. Everyone wins!

So much has happened in 2016 and January isn’t even over yet. The year is off to a great start!

Eric

Posted in Benchmark metrics, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, MobileXPRT, Performance benchmarking, TouchXPRT, TouchXPRT 2016, WebXPRT, What makes a good benchmark?, XPRT Weekly Tech Spotlight, XPRT Women Code-a-Thon |

Nebula Wolf

By Eric Hale

on December 10, 2015

A couple of months ago, we talked about the senior project we sponsored with North Carolina State University. We asked a small team of students to take a crack at implementing a game that we could use as the basis of a benchmark test.

Last Friday, the project culminated with a presentation at the annual Posters and Pies event.

The team gave a great presentation, and I was impressed by how much they accomplished in 3 months. They implemented a game that they called Nebula Wolf – the mascot for NC State is a wolf. It’s a space-themed rail shooter. You can play the game, or click a button to run a script for benchmarking purposes. In the scripted mode, Nebula Wolf unlocks the frame rate so the device can run at full speed.

Over the next couple of weeks, we’re going to be testing Nebula Wolf, digging into the code and getting a deeper understanding of what the team did. We’re hoping to make the game available on our web site soon.

Tomorrow, AJ, Brien, and Rachel will present one last time, here at PT. It’s been a real pleasure working with them. I wish them all good luck as they finish college and start their careers.

Eric

Posted in Benchmark metrics, Benchmarking, Collaborative benchmark development, Future of performance evaluation |

The TouchXPRT 2016 CP arrives tomorrow!

By Eric Hale

on November 12, 2015

As we said a couple of weeks ago, we wanted to test on Windows 10 Threshold 2 before releasing the TouchXPRT 2016 community preview. Well, Threshold 2 is out and the testing has been going very well.

We’ll release the TouchXPRT 2016 to the community tomorrow. Because community previews are not available to the general public, members will need to download it from our site.

The installation procedure is fairly straightforward. First, you put the device in developer mode. Then, for a tablet or PC, run a PowerShell script, as you did for TouchXPRT 2014. For a mobile device, once it’s in developer mode, copy the bundle to the device and install it.

If you have any problems, please let us know.

We’re looking forward to seeing the results from phones and, as we have done with MobileXPRT, to comparing results across different-sized devices.

Enjoy!

Eric

Posted in Benchmark metrics, Benchmarking, Collaborative benchmark development, Community Preview, TouchXPRT, TouchXPRT 2016, TouchXPRT development process, Windows 10 |

More than the sum of its parts

By Eric Hale

on October 22, 2015

There was a recent article in Bloomberg about phone maker ZTE’s increasing market share in the US. The article singled out one phone, the ZTE Maven, which costs about $60 (US).

This phrase jumped out at me: “a processor with capabilities somewhere between the iPhone 5 and 6.” The iPhone 5S could also fit that description. The ZTE Maven uses the ARM Cortex-A53, 64-bit processor running at 1.2 GHz. The Apple iPhone 5s uses the Apple Cyclone-A7 Cortex-A7 Harvard Superscalar processor running at 1.3 GHz.

We decided to put that statement to the test. We ran WebXPRT 2015 on the ZTE Maven and its score was 47. The iPhone 5s scored 100. The Maven was not even close.

As we’ve said before, the performance of a device depends on more than the GHz of its processor. For example, the ZTE Maven uses the Snapdragon 410 SoC, which was aimed at mid-level devices. The iPhone 5s uses the Apple A7, which was intended for higher-end devices. You can find side by side specs here.

Be wary when you see unsupported performance claims. As this example shows, specs can appear comparable even when the actual performance of the devices differs considerably. A good benchmark can provide insights into performance that specs alone can’t.

Eric

Posted in Android, Benchmark metrics, Benchmarking, WebXPRT, WebXPRT 2015 |

Category: Benchmark metrics

Focusing the spotlight

Is it hot in here?

The XPRT Women Code-a-Thon

Nebula Wolf

The TouchXPRT 2016 CP arrives tomorrow!

More than the sum of its parts

Check out the other XPRTs: