Category: What makes a good benchmark?

Personal preference

on May 19, 2016

I saw an interesting article recently, Here’s why I gave up my beloved Galaxy S7 for a boring old iPhone. It’s only been a few weeks since we featured the Samsung S7 in the XPRT Weekly Tech Spotlight, so of course I had to read it. The interesting thing is this guy really loved his Samsung S7, and even declared it “the best smartphone I’ve ever used.” He loved its VR capabilities, camera, and its look. He even prefers Android as an operating system.

So why would he give it up for an iPhone 6s Plus? Simply put, battery life. As a self-described heavy user, he found his Samsung S7 dying before 5 PM every day. The iPhone 6s Plus lasted much longer.

This is a good reminder that people have different priorities. Your priority could be having the fastest phone, the longest battery life, the best screen, or the broadest compatibility. This is why there is no such this as “the best device.”

This is why we are always asking for your input. Knowing your priorities helps the community build better tests!

Eric

Posted in Android, Battery life, BenchmarkXPRT development community, Collaborative benchmark development, Phones, What makes a good benchmark?, XPRT Weekly Tech Spotlight |

Feedback

By Eric Hale

on April 28, 2016

We’re excited by the high level of interest the community and vendors have shown in the upcoming cross-platform MobileXPRT benchmark. We’ve received general observations about what a cross-platform benchmark should be, along with detailed suggestions about tests, subsystems, and benchmark architecture. We appreciate all of the responses and welcome more, so please keep them coming!

The number-one concern we’ve heard is that we be sure the benchmark tests all platforms fairly. Transparency will be essential to assure users that the tests are performing the same work on all platforms and performing the work in the appropriate way for each platform.

Fortunately, the XPRTs are well positioned to address that concern. From the beginning, we have used a community model. The source code is available to all members, which is the ultimate in transparency. (If you’re not a community member, it’s easy to join!)

Speaking of source code, we released TouchXPRT source code to the community this week. Members can download the source here (login required).

Eric

Posted in Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Cross-platform benchmarks, Future of performance evaluation, MobileXPRT, Performance benchmarking, What makes a good benchmark? |

Women develop new perspectives for the XPRTs

By Jennie Faries

on March 18, 2016

Last weekend, we had the great privilege of co-hosting the first XPRT Women Code-a-Thon with the Seattle chapter of ChickTech. We couldn’t be happier with the results!

Our goal was to bring together a group of women and invite them to develop ideas for new device workloads—workloads that we might include in future versions of MobileXPRT and WebXPRT. The 20 participants—some working individually, and others working as teams—not only met that goal, they did a great deal more.

On the coding front, the participants achieved an impressive amount of work in a very short time. Though we awarded only three prizes, everyone generated interesting and useful ideas. Our prizes went to the following people:

1st place: Viveret Steele, for a 3D-modeling workload

2nd place: Annmarie Aidoo, for a geolocation workload

3rd place: Molly Fallen and Alex Trimble, for an audio-enhancement workload

These four people went home with checks, but winning wasn’t what motivated anyone to participate. Everyone was excited about developing software and working with others. The social side of the event proved to be as meaningful as the technical. People talked, formed friendships and mentoring relationships, and discussed seeking other events like this one. Two people said the event changed their lives.

In the weeks ahead, we’ll be sharing some more information about the event. In the meantime, we’re proud to have been part of it.

Jennie Faries

Last week in the XPRTs

We published the XPRT Weekly Tech Spotlight on the Microsoft Surface 3.
We added two new CrXPRT ’15 results.
We added two new MobileXPRT ’13 results.
We added six new WebXPRT ’15 results.

Posted in BenchmarkXPRT, BenchmarkXPRT development community, ChickTech Seattle, Collaborative benchmark development, Future of performance evaluation, MobileXPRT, WebXPRT, What makes a good benchmark?, XPRT Women Code-a-Thon |

Is it hot in here?

By Eric Hale

on March 3, 2016

One of the great meetings I had at CES was with another community member, Patrick Chang, Senior System Engineer in the Tablet Performance group at Dell. I was glad to hear that, when he tests his devices, Patrick makes frequent use of TouchXPRT.

While TouchXPRT stresses the system appropriately, Patrick’s job requires him to understand not only how well the device performs, but why it performs that way. He was wondering what we could do to help him correlate the temperature, power consumption, and performance of a device.

That information is not typically available to software and apps like the XPRTs. However, it may be possible to add some hooks that would let the XPRTs coordinate with third-party utilities and hardware that do.

As always, the input from the community guides the design of the XPRTs. So, we’d love to know how much interest the community has in having this type of information. If you have thoughts about this, or other kinds of information you’d like the XPRTs to gather, please let us know!

Eric

Last week in the XPRTs
We published the XPRT Weekly Tech Spotlight on the Apple iPad Pro.
We added one new BatteryXPRT ’14 result.
We added one new CrXPRT ’15 result.
We added one new MobileXPRT ’13 result.
We added four new WebXPRT ’15 results.

Posted in Benchmark metrics, Benchmarking, BenchmarkXPRT, CES, Collaborative benchmark development, Dell, Future of performance evaluation, Patrick Chang, Performance benchmarking, Power consumption, Temperature, TouchXPRT, Trade Shows, What makes a good benchmark? |

A clarification from Brett Howse

By Eric Hale

on February 18, 2016

A couple of weeks ago, I described a conversation I had with Brett Howse of AnandTech. Brett was kind enough to send a clarification of some of his remarks, which he gave us permission to share with you.

“We are at a point in time where the technology that’s been called mobile since its inception is now at a point where it makes sense to compare it to the PC. However we struggle with the comparisons because the tools used to do the testing do not always perform the same workloads. This can be a major issue when a company uses a mobile workload, and a desktop workload, but then puts the resulting scores side by side, which can lead to misinformed conclusions. This is not only a CPU issue either, since on the graphics side we have OpenGL well established, along with DirectX, in the PC space, but our mobile workloads tend to rely on OpenGL ES, with less precision asked of the GPU, and GPUs designed around this. Getting two devices to run the same work is a major challenge, but one that has people asking what the results would be.”

I really appreciate Brett taking the time to respond. What are your thoughts in these issues? Please let us know!

Eric

Posted in AnandTech, Collaborative benchmark development, DirectX, Future of performance evaluation, Let us know your thoughts, Mobile devices, Open GL, Performance benchmarking, Performance of computing devices, Performance testing on tablets, What makes a good benchmark? |

Comparing apples and oranges?

By Eric Hale

on February 4, 2016

My first day at CES, I had breakfast with Brett Howse from AnandTech. It was a great opportunity to get the perspective of a savvy tech journalist and frequent user of the XPRTs.

During our conversation, Brett raised concerns about comparing mobile devices to PCs. As mobile devices get more powerful, the performance and capability gaps between them and PCs are narrowing. That makes it more common to compare upper-end mobile devices to PCs.

People have long used different versions of benchmarks when comparing these two classes of devices. For example, the images for benchmarking a phone might be smaller than those for benchmarking a PC. Also, because of processor differences, the benchmarks might be built differently, say a 16- or 32-bit executable for a mobile device, and a 64-bit version for a PC. That was fine when no one was comparing the devices directly, but can be a problem now.

This issue is more complicated than it sounds. For those cases where a benchmark uses a dumbed-down version of the workload for mobile devices, comparing the results is clearly not valid. However, let’s assume that the workload stays the same, and that you run a 32-bit benchmark on a tablet, and a 64-bit version on a PC. Is the comparison valid? It may be, if you are talking about the day-to-day performance a user is likely to encounter. However, it may not be valid if you are making statement about the potential performance of the device itself.

Brett would like the benchmarking community to take charge of this issue and provide guidance about how to compare mobile devices and PCs. What are your thoughts?

Eric

Posted in AnandTech, BenchmarkXPRT development community, Brett Howse, CES, Collaborative benchmark development, Future of performance evaluation, Mobile devices, PCs, Performance of computing devices, Performance testing on tablets, What makes a good benchmark? |

Category: What makes a good benchmark?

Personal preference

Feedback

Women develop new perspectives for the XPRTs

Is it hot in here?

A clarification from Brett Howse

Comparing apples and oranges?

Check out the other XPRTs: