Category: CES

Quarterly review

on March 31, 2016

It’s been one of our busiest quarters ever! Here’s a quick review of what’s been happening:

The XPRTs were on the road a lot!

In January, I was at CES in Las Vegas.
In February, Mark was at MWC in Barcelona.
In March, we held the first ever XPRT Women Code-A-Thon in Seattle.

While I was at CES, I was lucky enough to be able to sit down and talk on-the-record with a couple of community members:

Brett Howse of AnandTech, who also wrote a clarification of some points from our conversation.
Patrick Chang, Senior System Engineer in the Tablet Performance group at Dell.

Many thanks to them for being so generous with their time and their insights.

We also gave folks a lot to look at:

TouchXPRT 2016: After solving a problem with TouchXPRT’s Create Slideshow test and releasing a new community preview, we released it to the general public.
Nebula Wolf: We sponsored a semester-long senior project at North Carolina State University, which resulted in a game-based workload. There’s a really cool video, and community members can play the game (login required).
XPRT Weekly Tech Spotlight: Every week we put a different device in the spotlight, giving its specs and testing it with the XPRTs.

That is a great start to the year, but we’re going to top it – Next week, we’ll kick off Q2 with one of our biggest announcements ever!

Eric

Posted in AnandTech, BenchmarkXPRT, Brett Howse, CES, Collaborative benchmark development, Community Preview, Dell, Mobile World Congress, Mobile World Congress, Nebula Wolf, North Carolina State University, Patrick Chang, TouchXPRT, TouchXPRT 2016, XPRT Women Code-a-Thon |

One of the great meetings I had at CES was with another community member, Patrick Chang, Senior System Engineer in the Tablet Performance group at Dell. I was glad to hear that, when he tests his devices, Patrick makes frequent use of TouchXPRT.

While TouchXPRT stresses the system appropriately, Patrick’s job requires him to understand not only how well the device performs, but why it performs that way. He was wondering what we could do to help him correlate the temperature, power consumption, and performance of a device.

That information is not typically available to software and apps like the XPRTs. However, it may be possible to add some hooks that would let the XPRTs coordinate with third-party utilities and hardware that do.

As always, the input from the community guides the design of the XPRTs. So, we’d love to know how much interest the community has in having this type of information. If you have thoughts about this, or other kinds of information you’d like the XPRTs to gather, please let us know!

Eric

Last week in the XPRTs
We published the XPRT Weekly Tech Spotlight on the Apple iPad Pro.
We added one new BatteryXPRT ’14 result.
We added one new CrXPRT ’15 result.
We added one new MobileXPRT ’13 result.
We added four new WebXPRT ’15 results.

Posted in Benchmark metrics, Benchmarking, BenchmarkXPRT, CES, Collaborative benchmark development, Dell, Future of performance evaluation, Patrick Chang, Performance benchmarking, Power consumption, Temperature, TouchXPRT, Trade Shows, What makes a good benchmark? |

Comparing apples and oranges?

By Eric Hale

on February 4, 2016

My first day at CES, I had breakfast with Brett Howse from AnandTech. It was a great opportunity to get the perspective of a savvy tech journalist and frequent user of the XPRTs.

During our conversation, Brett raised concerns about comparing mobile devices to PCs. As mobile devices get more powerful, the performance and capability gaps between them and PCs are narrowing. That makes it more common to compare upper-end mobile devices to PCs.

People have long used different versions of benchmarks when comparing these two classes of devices. For example, the images for benchmarking a phone might be smaller than those for benchmarking a PC. Also, because of processor differences, the benchmarks might be built differently, say a 16- or 32-bit executable for a mobile device, and a 64-bit version for a PC. That was fine when no one was comparing the devices directly, but can be a problem now.

This issue is more complicated than it sounds. For those cases where a benchmark uses a dumbed-down version of the workload for mobile devices, comparing the results is clearly not valid. However, let’s assume that the workload stays the same, and that you run a 32-bit benchmark on a tablet, and a 64-bit version on a PC. Is the comparison valid? It may be, if you are talking about the day-to-day performance a user is likely to encounter. However, it may not be valid if you are making statement about the potential performance of the device itself.

Brett would like the benchmarking community to take charge of this issue and provide guidance about how to compare mobile devices and PCs. What are your thoughts?

Eric

Posted in AnandTech, BenchmarkXPRT development community, Brett Howse, CES, Collaborative benchmark development, Future of performance evaluation, Mobile devices, PCs, Performance of computing devices, Performance testing on tablets, What makes a good benchmark? |

In the spotlight

By Eric Hale

on January 14, 2016

I’m happy to be back in North Carolina, but I had a really great time at CES. I talked to over a dozen companies about the XPRTs and the XPRT Weekly Tech Spotlight, and had some good conversations. Hopefully, some of these companies’ devices will be among the first ones we showcase when the XPRT Weekly Tech Spotlight goes live next month.

Of course, I saw some really great tech at CES! Amazing TVs and cars, magic mirrors, all kinds of drones, and the list goes on. Before the show, the Internet of Things was predicted to be big this year, and boy, was it! Smart refrigerators, door locks, and thermostats were just the beginning. Some of my favorite examples were the chopsticks and footbath—both Bluetooth enabled—and “the world’s first remote controlled game shoe.”

Clearly IoT is the Wild West of technology right now. We’re had some conversations about how the XPRTs might be able to help consumers navigate the chaos. However, with a class of products this diverse, there are a lot of issues to consider. If you have any thoughts about this, let us know!

Eric

Posted in CES, Collaborative benchmark development, Future of performance evaluation, Internet of things, IoT, What makes a good benchmark?, XPRT Weekly Tech Spotlight |

CES day 1

By Eric Hale

on January 7, 2016

It’s Wednesday night as I’m writing this. CES started in earnest today, and I had a really great day. I saw a lot of cool technology. I really liked the TVs. As I tweeted earlier today, I’m not sure that real life looks that good! That being said, the Ferrari was probably my favorite.

I’m not yet convinced about having the family calendar on a tablet embedded in the refrigerator. However, several earnest people made a very good case for it.

This morning PT announced a new program called the XPRT Weekly Tech Spotlight. This is a new way for device vendors and manufacturers to share verified results with buyers around the world. I was happy to talk with a number of different vendors about it and about the XPRTs.

I’ve also met with some community members while I’ve been here. It’s always great to hear from the community, but there’s nothing like being able to sit and talk. The conversations raised some interesting issues, and I’ll be writing about them in future blog posts.

I’m really looking forward to the rest of the show!

Eric

Posted in BenchmarkXPRT, BenchmarkXPRT development community, CES, Collaborative benchmark development, XPRT Weekly Tech Spotlight |

Another great year

By Eric Hale

on December 31, 2015

A lot of great stuff happened this year! In addition to releasing new versions of the benchmarks, videos, infographics, and white papers, we released our first-ever German UI and sponsored our first student partnership at North Carolina State University. We visited three continents to promote the XPRTs and saw XPRT results published in six of them (we’re still working on Antarctica).

Perhaps most exciting, we reached our fifth anniversary. Users have downloaded or run the XPRTs over 100,000 times.

As great as the year has been, we are sprinting into 2016. Though I can’t talk about them yet, there are some big pieces of news coming soon. Even sooner, I will be at CES next week. If you would like to talk about the XPRTs or the future of benchmarking, let me know and we’ll find a time to meet.

Whatever your holiday traditions are, I hope you are having a great holiday season. Here’s wishing you all the best in 2016!

Eric

Posted in Benchmarking, Benchmarks in general, BenchmarkXPRT development community, CES, Collaborative benchmark development, Future of performance evaluation, German, History of benchmarking |

Category: CES

Quarterly review

Is it hot in here?

Comparing apples and oranges?

In the spotlight

CES day 1

Another great year

Check out the other XPRTs: