Category: iOS

Exploring virtual reality

on December 15, 2016

We’ve talked a lot in recent weeks about new technologies we are evaluating for the XPRTs. You may remember that back in June, we also wrote about sponsoring a second senior project with North Carolina State University. Last week, the project ended with the traditional Posters and Pies event. The team gave a very well thought‑out presentation.

As you can tell from the photo below, the team designed and implemented a nifty virtual reality app. It’s a room escape puzzle, and it looks great!

The app is a playable game with the ability to record the gameplay for doing repeatable tests. It also includes a recording that allows you to test a device without playing the game. Finally, the app lets you launch directly into the prerecorded game without using a viewer, which will be handy for testing multiple devices.

The team built the app using the Google Cardboard API and the Unity game engine, which allowed them to create Android and iOS versions. We’re looking forward to seeing what that may tell us!

After Posters and Pies, the team came to PT to present their work and answer questions. We were all very impressed with their knowledge and with how well thought out the application was.

Many thanks to team members Christian McCurdy, Gregory Manning, Grayson Jones, and Shon Ferguson (not shown).

Thanks also to Dr. Lina Battestilli, the team’s technical advisor, and Margaret Heil, Director of the Senior Design Center.

We are currently evaluating the app, and expect to make it available to the community in early 2017!

Eric

Posted in Android, Benchmarking, Collaborative benchmark development, iOS, North Carolina State University, Virtual reality |

Tracking device evolution with WebXPRT ’15, part 2

By Justin Greene

on October 27, 2016

Last week, we used the Apple iPhone as a test case to show how hardware advances are often reflected in benchmark scores over time. When we compared WebXPRT 2015 scores for various iPhone models, we saw a clear trend of progressively higher scores as we moved from phones with an A7 chip to phones with A8, A9, and A10 Fusion chips. Performance increases over time are not surprising, but WebXPRT ’15 scores also showed us that upgrading from an iPhone 6 to an iPhone 6s is likely to have a much greater impact on web-browsing performance than upgrading from an iPhone 6s to an iPhone 7.

This week, we’re revisiting our iPhone test case to see how software updates can boost device performance without any changes in hardware. The original WebXPRT ’15 tests for the iPhone 5s ran on iOS 8.3, and the original tests for the iPhone 6s, 6s Plus, and SE ran on variants of iOS 9. We updated each phone to iOS 10.0.2 and ran several iterations of WebXPRT ’15.

Upgrading from iOS 8.3 to iOS 10 on the iPhone 5s caused a 17% increase in web-browsing performance, as measured by WebXPRT. Upgrading from iOS 9 to iOS 10 on the iPhone 6s, 6s Plus, and SE produced web-browsing performance gains of 2.6%, 3.6%, and 3.1%, respectively.

The chart below shows the WebXPRT ’15 scores for a range of iPhones, with each iPhone’s iOS version upgrade noted in parentheses. The dark blue columns on the left represent the original scores, and the light blue columns on the right represent the upgrade scores.

As with our hardware comparison last week, these scores are the median of a range of scores for each device in our database. These scores come both from our own testing and from device reviews from popular tech media outlets.

These results reinforce a message that we repeat often, that many factors other than hardware influence performance. Designing benchmarks that deliver relevant and reliable scores requires taking all factors into account.

What insights have you gained recently from WebXPRT ’15 testing? Let us know!

Justin

Posted in Apple, Application-based benchmarks, Benchmark metrics, Benchmarking, Browser-based benchmarks, iOS, Performance benchmarking, Uncategorized, WebXPRT |

Getting it right

By Eric Hale

on June 2, 2016

Back in April Bill announced that we are working on a cross-platform benchmark. We asked for your thoughts and comments, and you’ve been great! We really appreciate all the great ideas.

We’ve been using code from MobileXPRT and TouchXPRT as the basis for some experiments. In his post, Bill talked about the difficulty of porting applications. However, even though we have expertise in porting applications, it’s proving more difficult than we originally thought. Benchmarks are held to a higher standard than most applications. It’s not enough for the code to run reliably and efficiently, it must compare the different platforms fairly.

One thing we know for sure: getting it right is going to take a while. However, we owe it to you to make sure that the benchmark is reliable and fair on all platforms it supports. We will, of course, keep you informed as things progress.

In the meantime, keep sending your ideas!
Eric

Posted in Android, Application-based benchmarks, Benchmark metrics, Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, iOS, MobileXPRT, Performance benchmarking, TouchXPRT, WebXPRT, Windows 10 |

Looking ahead

By Eric Hale

on April 21, 2016

It’s only been a couple of weeks since we announced a cross-platform XPRT. It’s still early days, but we’ve already started getting ideas from vendors and media—from both people within the community and those who have not yet joined. We’re incorporating these ideas into our investigations, and plan to be sending a design document for the community to critique in a few weeks.

However, we are always looking ahead and Bill’s trip to IDF16 got us thinking about future benchmarks. Virtual reality is obviously going to be big. Bill said that he thinks he saw more things using the Oculus Rift than there are Oculus Rifts in the world! The Internet of Things has been ramping up for a while now, and shows no sign of slowing down. Computer vision is another emerging area, one with many possible applications. There are a lot of exciting possibilities!

As always, we want to know what you think. What upcoming technologies are you excited about? What would like to see in these benchmarks? Please let us know!

Eric

Posted in Android, Benchmark metrics, BenchmarkXPRT development community, Collaborative benchmark development, Cross-platform benchmarks, Future of performance evaluation, IDF, iOS, MobileXPRT, Virtual reality, Windows 10 |

One benchmark to test them all

By Bill Catchings

on April 7, 2016

It’s no secret that the XPRTs are a great way to get device results you can count on. Tens of thousands of people over six continents have used the XPRTs to help them make smart buying choices, and over a thousand media outlets have quoted XPRT results when reporting on the hottest tech. WebXPRT has always been the “go to” XPRT, because you can use it to test the widest range of devices. WebXPRT runs in the browser, however, so browser performance influences the results.

For a long time, our members and others have asked for a tool that would let you compare application performance on any type of device. People want a cross-platform XPRT that runs on devices the same way apps do.

We’re excited to announce that we’re going to create just that tool! Specifically, we’re going to create a version of MobileXPRT that runs on Android, iOS, and Windows.

This will not be easy. At one point in my career, I was in charge of a group that ported applications between platforms, and I learned from hands-on experience that doing that job well is very difficult. It’s not enough to simply make the application run; it also has to run efficiently on each type of system. MobileXPRT works at the application level, so we’ll have to deal with the many differences in the operating system architectures and APIs. We’ll have to make sure the code runs well on all three target OSes.

We’re willing to do all this work because the need for such a tool has never been greater. More and more devices hit the market all the time, and choosing the ones you want is tougher than ever. iPhone or Android phone? Windows tablet, Android tablet, or iPad?

The coming MobileXPRT will let buyers around the world answer those questions.

We’re not going to do this work in isolation. We will reach out to the OS vendors, because we want their input, comments, and help. We’ll make the source available to them, and we welcome their critiques and guidance in creating the best possible version for each OS.

Of course, we very much want your input, too. Do you have any thoughts about what you’d like to see in a cross-platform XPRT? If so, let us know!

Bill

Last week in the XPRTs
We published the XPRT Weekly Tech Spotlight on the Apple iPhone SE.
We added one new MobileXPRT ’15 result.
We added seven new WebXPRT ’15 results.