Category: Future of performance evaluation

Evolve or die

on April 20, 2017

Last week, Google announced that it would retire its Octane benchmark. Their announcement explains that they designed Octane to spur improvement in JavaScript performance, and while it did just that when it was first released, those improvements have plateaued in recent years. They also note that there are some operations in Octane that optimize Octane scores but do not reflect real-world scenarios. That’s unfortunate, because they, like most of us, want improvements in benchmark scores to mean improvements in end-user experience.

WebXPRT comes at the web performance issue differently. While Octane’s goal was to improve JavaScript performance, the purpose of WebXPRT is to measure performance from the end user’s perspective. By doing the types of work real people do, WebXPRT doesn’t measure only improvements in JavaScript performance; it also measures the quality of the real-world user experience. WebXPRT’s results also reflect the performance of the entire device and software stack, not just the performance of the JavaScript interpreter.

Google’s announcement reminds us that benchmarks have finite life spans, that they must constantly evolve to keep pace with changes in technology, or they will become useless. To make sure the XPRT benchmarks do just that, we are always looking at how people use their devices and developing workloads that reflect their actions. This is a core element of the XPRT philosophy.

As we mentioned last week, we’ve working on the next version of WebXPRT. If you have any thoughts about how it should evolve, let us know!

Eric

Posted in Benchmark metrics, Benchmarking, BenchmarkXPRT development community, Browser-based benchmarks, Future of performance evaluation, Google, JavaScript, Performance benchmarking, What makes a good benchmark? |

Thinking ahead to WebXPRT 2017

By Justin Greene

on April 13, 2017

A few months ago, Bill discussed our intention to update WebXPRT this year. Today, we want to share some initial ideas for WebXPRT 2017 and ask for your input.

Updates to the workloads provide an opportunity to increase the relevance and value of WebXPRT in the years to come. Here are a few of the ideas we’re considering:

For the Photo Enhancement workload, we can increase the data sizes of pictures. We can also experiment with additional types of photo enhancement such as background/foreground subtraction, collage creation, or panoramic/360-degree image viewing.
For the Organize Album workload, we can explore machine learning workloads by incorporating open source JavaScript libraries into web-based inferencing tests.
For the Local Notes workload, we’re investigating the possibility of leveraging natural-brain libraries for language processing functions.
For a new workload, we’re investigating the possibility of using online 3D modeling applications such as Tinkercad.

For the UI, we’re considering improvements to features like the in-test progress bars and individual subtest selection. We’re also planning to update the UI to make it visually distinct from older versions.

Throughout this process, we want to be careful to maintain the features that have made WebXPRT our most popular tool, with more than 141,000 runs to date. We’re committed to making sure that it runs quickly and simply in most browsers and produces results that are useful for comparing web browsing performance across a wide variety of devices.

Do you have feedback on these ideas or suggestions for browser technologies or test scenarios that we should consider for WebXPRT 2017? Are there existing features we should ditch? Are there elements of the UI that you find especially useful or would like to see improved? Please let us know. We want to hear from you and make sure that we’re crafting a performance tool that continues to meet your needs.

Justin

Posted in BenchmarkXPRT development community, Browser-based benchmarks, Collaborative benchmark development, Future of performance evaluation, Let us know your thoughts, Machine learning, Performance benchmarking, WebXPRT, WebXPRT 2017, What makes a good benchmark? |

Mobile World Congress 2017 and the territories ahead

By Mark Van Name

on March 2, 2017

Walking the halls of this year’s Mobile World Congress (MWC)—and, once again, I walked by every booth in every one of them—it was clear that mobile technology is expanding faster than ever into more new tech territories than ever before.

On the device front, cameras and camera quality have become a pitched battleground, with mobile phone makers teaming with camera manufacturers to give us better and better images and video. This fight is far from over, too, because vendors are exploring many different ways to improve mobile phone camera quality. Quick charging is a hot new trend we can expect to hear more about in the days to come. Of course, apps and their performance continue to matter greatly, because if you can do it from any computer, you better be able to do at least some of it from your phone.

The Internet of Things (IoT) grabbed many headlines, with vendors still selling more dreams than reality, but some industries living this future now. The proliferation of IoT devices will result, of course, in massive increases in the amount of data flowing through the world’s networks, which in turn will require more and more computing power to analyze and use. That power will need to be everywhere, from massive datacenters to the device in your hand, because the more data you have, the more you’ll want to customize it to your particular needs.

Similarly, AI was a major theme of the show, and it’s also likely to suck up computing cycles everywhere. The vast majority of the work will, of course, end up in datacenters, but some processing is likely to be local, particularly in situations, such as real-time translation, where we can’t afford significant comm delays.

5G, the next big step in mobile data speeds, was everywhere, with most companies seeming to agree the new standard was still years away–but also excited about what will be possible. When you can stream 4K movies to your phone wirelessly while simultaneously receiving and customizing analyses of your company’s IoT network, you’re going to need a powerful, sophisticated device running equally powerful and sophisticated apps.

Everywhere I looked, the future was bright—and complicated, and likely to place increasing demands on all of our devices. We’ll need guides as we find our paths through these new territories and as we determine the right device tools for our jobs, so the need for the XPRTs will only increase. I look forward to seeing where we, the BenchmarkXPRT Development Community, take them next.

Mark

Posted in 4K, 5G, AI, Future of performance evaluation, Internet of things, IoT, Mobile data, Mobile devices, Mobile World Congress, Mobile World Congress, Phones, Trade Shows |

A new reality

By Eric Hale

on February 23, 2017

A while back, I wrote about a VR demo built by students from North Carolina State University. We’ve been checking it out over the last couple of months and are very impressed. This workload will definitely heat up your device! While the initial results look promising, this is still an experimental workload and it’s too early to use results in formal reviews or product comparisons.

We’ve created a page that tells all about the VR demo. As an experimental workload, the demo is available only to community members. As always, members can download the source as well as the APK.

We asked the students to try to build the workload for iOS as a stretch goal. They successfully built an iOS version, but this was at the end of the semester and there was little time for testing. If you want to experiment with iOS yourself, look at the build instructions for Android and iOS that we include with the source. Note that you will need Xcode to build and deploy the demo on iOS.

After you’ve checked out the workload, let us know what you think!

Finally, we have a new video featuring the VR demo. Enjoy!

Eric

Posted in Application-based benchmarks, BenchmarkXPRT development community, Collaborative benchmark development, Education, Future of performance evaluation, Mobile devices, North Carolina State University, Virtual reality, VR Demo, What makes a good benchmark? |

TouchXPRT’s future

By Bill Catchings

on February 2, 2017

If you’ve been following the blog, you know that we’ve been reviewing each part of the XPRT portfolio. If you missed our discussions of HDXPRT, BatteryXPRT, WebXPRT, and CrXPRT, we encourage you to check them out and send us any thoughts you may have. This week, we continue that series by discussing the state of TouchXPRT and what we see down the road for it in 2017.

We released TouchXPRT 2016, an app for evaluating the performance of Windows 10 and Windows 10 Mobile devices, last February. We built the app by porting TouchXPRT 2014 performance workloads to the new Universal Windows App format, which allows a single app package to run on PCs, phones, tablets, and even consoles.

TouchXPRT 2016 installation is quick and easy, and the test completes in under 15 minutes on most devices. The app runs tests based on five everyday tasks (Beautify Photos, Blend Photos, Convert Videos for Sharing, Create Music Podcast, and Create Slideshow from Photos). It measures how long your device takes to complete each task, produces results for each scenario, and gives you an overall score.

As we think about the path forward for TouchXPRT, we’re aware that many expect 2017 to be a year of significant change in the Windows world, with two updates scheduled for release. Microsoft is slated to release the Windows 10 Creators Update (Build 1704) in April, and a subsequent version of Windows codenamed Redstone 3 may arrive this fall. Many tech observers believe that the Creators Update will introduce new creativity and gaming features, along with a UI upgrade named Project NEON. Major foundational shifts in the OS’s structure are more likely to appear with Redstone 3. At this point, quite a lot is still up in the air, but we’ll be following developments closely.

As we learn more about upcoming changes, we’ll have the opportunity to reevaluate TouchXPRT workloads and determine the best way to incorporate new technologies. Virtual reality, 3D, and 4K are especially exciting, but it’s too soon to know how we might incorporate them in a future version of TouchXPRT.

Because TouchXPRT 2016 continues to run well on a wide range of Windows 10 devices, we think it’s best to keep supporting the current version until we get a better idea of what’s in store for Windows.

If you have any thoughts on the future of Windows performance testing, please let us know!

Bill

Posted in BatteryXPRT 2014 for Android, BenchmarkXPRT development community, CrXPRT, Future of performance evaluation, HDXPRT, MobileXPRT, PCs, TouchXPRT, TouchXPRT 2016, TouchXPRT development process, TouchXPRT release cycle, WebXPRT 2017, Windows 10 |

CES 2017

By Bill Catchings

on January 6, 2017

I’ve attended many tech shows over the years, but this year’s CES has more energy than any I’ve attended in a long time. Part of the energy is the breadth of products. There are amazingly slim TVs that make my TV at home, which I thought was slim, look fat. And, there are beautiful 8K TVs that make my new 4K one feel old.

I’m seeing all manner of smartphones. I’m seeing mobile remote presence devices such as the one from Beam, and after seeing the latest Fenix 5 from Garmin, I think I’ve found my next smartwatch.

There are many differing devices and approaches to VR and AR. There are drones everywhere. And, lots of massage chairs.

There are also plenty of products, let’s call them longshots, that contribute to the Wild West feel of the show. Maybe you’d like the Hydreon FakeTV, a small device to make it seem like a TV is on in your house to keep away burglars? Or Dr. Fuji’s Body Shaper, a vibrating platform to “accelerate your workout”?

Another part of the dynamic feeling is the breadth of vendors. Almost all the big tech vendors are here, except for Apple, of course. The excitement for me, however, is the small vendors displaying things that may well never see the light of day, but give glimpses of the future. For example, while I doubt most of the drone vendors at CES will be around in a few years, I think the trend toward small, inexpensive selfie drones will be.

The main reason for the energy at this year’s CES could be the convergence of multiple big industries. The most obvious example of this phenomenon is the large auto presence. Cars have been at CES before—the first time I drove my current car, a BMW i3, was at CES 2014—but this time around they seem to really want to make a statement. Faraday Future is making a splash by trying to be the next Tesla with its FF91.

Multiple vendors, including VW, Nissan, Toyota, and Mercedes, have concept cars on display—most of them are electric and all of them are heavy on technology. The biggest tech they’re touting is autonomous driving. The auto companies are showing their products while companies like NVIDIA, Intel, and Magic Eye are displaying the tech they have as well.

Regardless of the source of the energy at this CES, I see many opportunities for the existing XPRTs to continue to be important resources. I also see how important emerging technologies like machine learning and VR/AR are going to be and how the XPRTs can be of help there as well.

Exciting times!