BenchmarkXPRT Blog banner

Category: Benchmarking

Feedback

We’re excited by the high level of interest the community and vendors have shown in the upcoming cross-platform MobileXPRT benchmark. We’ve received general observations about what a cross-platform benchmark should be, along with detailed suggestions about tests, subsystems, and benchmark architecture. We appreciate all of the responses and welcome more, so please keep them coming!

The number-one concern we’ve heard is that we be sure the benchmark tests all platforms fairly. Transparency will be essential to assure users that the tests are performing the same work on all platforms and performing the work in the appropriate way for each platform.

Fortunately, the XPRTs are well positioned to address that concern. From the beginning, we have used a community model. The source code is available to all members, which is the ultimate in transparency.  (If you’re not a community member, it’s easy to join!)

Speaking of source code, we released TouchXPRT source code to the community this week. Members can download the source here (login required).

Eric

Looking ahead

It’s only been a couple of weeks since we announced a cross-platform XPRT. It’s still early days, but we’ve already started getting ideas from vendors and media—from both people within the community and those who have not yet joined. We’re incorporating these ideas into our investigations, and plan to be sending a design document for the community to critique in a few weeks.

However, we are always looking ahead and Bill’s trip to IDF16 got us thinking about future benchmarks. Virtual reality is obviously going to be big. Bill said that he thinks he saw more things using the Oculus Rift than there are Oculus Rifts in the world! The Internet of Things has been ramping up for a while now, and shows no sign of slowing down. Computer vision is another emerging area, one with many possible applications. There are a lot of exciting possibilities!

As always, we want to know what you think. What upcoming technologies are you excited about? What would like to see in these benchmarks? Please let us know!

Eric

IDF16 Shenzhen

I just spent the last couple of days at IDF16 Shenzhen. It was a great opportunity to talk to folks about the XPRTs, see some future technology demos, and think about the future of the XPRTs.

The technology and product demos included lots of interesting technology. I saw everything from the latest computers to games to VR to body monitoring.

IDF16-1

Of particular interest to me were the future-looking technologies beyond the usual array of notebooks, tablets, and servers. I was able to see drones that could video a person by following them, while avoiding obstacles such as trees. I saw a number of demos using the Oculus Rift. I got to see some robot demos that were impressive in their use of the fairly off-the-shelf technology driving them. I would have had myself scanned and then had a small 3D model of myself printed, but I was pressed for time and the line was too long.

I was particularly interested in a mirror that could scan a person and tell things about their health. I also found somewhat amusing a technology demo that was able to “beautify” a person in real time for use with teleconferencing such as Skype. While I might quibble about the definition of beautify, the idea of real-time video enhancement is intriguing. (Given the raw material I gave it to work with, it was no easy task to accomplish!) Maybe I won’t need to shave before my next WebEx meeting…

IDF16-2

All of these technologies give some hints as to areas the XPRTs may go in the future. While I don’t think we are quite ready for BeautificationXPRT, there may well be some workloads we should consider such as path finding, real-time video enhancement, health monitoring, virtual reality, and gaming. Please let us know your thoughts about what near-term technologies we should be considering in future workloads.

We definitely have exciting times still ahead of us in technology and the XPRTs!

Bill

One benchmark to test them all

It’s no secret that the XPRTs are a great way to get device results you can count on. Tens of thousands of people over six continents have used the XPRTs to help them make smart buying choices, and over a thousand media outlets have quoted XPRT results when reporting on the hottest tech. WebXPRT has always been the “go to” XPRT, because you can use it to test the widest range of devices. WebXPRT runs in the browser, however, so browser performance influences the results.

For a long time, our members and others have asked for a tool that would let you compare application performance on any type of device. People want a cross-platform XPRT that runs on devices the same way apps do.

We’re excited to announce that we’re going to create just that tool! Specifically, we’re going to create a version of MobileXPRT that runs on Android, iOS, and Windows.

This will not be easy. At one point in my career, I was in charge of a group that ported applications between platforms, and I learned from hands-on experience that doing that job well is very difficult. It’s not enough to simply make the application run; it also has to run efficiently on each type of system. MobileXPRT works at the application level, so we’ll have to deal with the many differences in the operating system architectures and APIs. We’ll have to make sure the code runs well on all three target OSes.

We’re willing to do all this work because the need for such a tool has never been greater. More and more devices hit the market all the time, and choosing the ones you want is tougher than ever. iPhone or Android phone? Windows tablet, Android tablet, or iPad?

The coming MobileXPRT will let buyers around the world answer those questions.

We’re not going to do this work in isolation. We will reach out to the OS vendors, because we want their input, comments, and help. We’ll make the source available to them, and we welcome their critiques and guidance in creating the best possible version for each OS.

Of course, we very much want your input, too. Do you have any thoughts about what you’d like to see in a cross-platform XPRT? If so, let us know!

Bill

Last week in the XPRTs
We published the XPRT Weekly Tech Spotlight on the Apple iPhone SE.
We added one new MobileXPRT ’15 result.
We added seven new WebXPRT ’15 results.

Quarterly review

It’s been one of our busiest quarters ever! Here’s a quick review of what’s been happening:

The XPRTs were on the road a lot!

 
While I was at CES, I was lucky enough to be able to sit down and talk on-the-record with a couple of community members:

 
Many thanks to them for being so generous with their time and their insights.

We also gave folks a lot to look at:

 
That is a great start to the year, but we’re going to top it – Next week, we’ll kick off Q2 with one of our biggest announcements ever!

Eric

Focusing the spotlight

As you may have heard, the Samsung Galaxy S7 is the XPRT Weekly Tech Spotlight this week.  As we were testing it, we noticed that our WebXPRT scores were about 8 percent lower than those reported by AnandTech.

The folks at AnandTech do a good job on their reviews, so we wanted to understand the discrepancy in scores. The S7 comes in a couple of models, so we started by verifying that our model was the same as theirs. It was.

The next step was to check their configuration against ours, and this is where we found the difference. Both phones were running the same version of Android, but the S7 AnandTech tested used Chrome 48 while the S7 we tested came preloaded with Chrome 49. In our testing, we’ve noticed that upgrading from Chrome 48 to Chrome 49 has a noticeable performance impact on certain devices. On the Samsung Galaxy S6, the scores went down about 10 percent. In all cases we’ve seen, the decrease is driven largely by the Stock Option Pricing workload.

This isn’t the first time we’ve written about browser versions affecting results. WebXPRT is a browsing benchmark, and the browser has a legitimate impact on performance. When you’re comparing results, it’s always important to look at all the factors involved.

Justin

Last week in the XPRTs

We published the XPRT Weekly Tech Spotlight on the Samsung Galaxy S7.
We added two new BatteryXPRT ’14 results.
We added one new MobileXPRT ’15 result.
We added four new WebXPRT ’15 results.

Check out the other XPRTs:

Forgot your password?