Category: Collaborative benchmark development

A new reality

on February 23, 2017

A while back, I wrote about a VR demo built by students from North Carolina State University. We’ve been checking it out over the last couple of months and are very impressed. This workload will definitely heat up your device! While the initial results look promising, this is still an experimental workload and it’s too early to use results in formal reviews or product comparisons.

We’ve created a page that tells all about the VR demo. As an experimental workload, the demo is available only to community members. As always, members can download the source as well as the APK.

We asked the students to try to build the workload for iOS as a stretch goal. They successfully built an iOS version, but this was at the end of the semester and there was little time for testing. If you want to experiment with iOS yourself, look at the build instructions for Android and iOS that we include with the source. Note that you will need Xcode to build and deploy the demo on iOS.

After you’ve checked out the workload, let us know what you think!

Finally, we have a new video featuring the VR demo. Enjoy!

Eric

Posted in Application-based benchmarks, BenchmarkXPRT development community, Collaborative benchmark development, Education, Future of performance evaluation, Mobile devices, North Carolina State University, Virtual reality, VR Demo, What makes a good benchmark? |

A new HDXPRT 2014 build is available

By Justin Greene

on February 9, 2017

Last fall, we identified a way to run HDXPRT 2014, originally developed for Windows 8, on Windows 10. The method involved overwriting the HDXPRT CPU-Z files with newer versions and performing a few additional pre-test configuration steps. You can read more details about those steps here.

Today, we’re releasing a new build of HDXPRT 2014 (v1.2) that eliminates the need to overwrite the CPU-Z files. The new build is available for download at HDXPRT.com. Please note that the app package is 5.08 GB, so allow time and space for the download process.

We also updated the HDXPRT 2014 User Manual to reflect changes in pre-test system configuration and to include the settings we recommend for newer builds of Windows 10.

The changes in the new build do not affect results, so v1.2 scores are comparable to v1.1 scores on the same system.

The new build ran well during testing in our labs, but issues could emerge as Microsoft releases new Windows updates. If you have any questions about HDXPRT or encounter any issues during testing, we encourage you to let us know.

We look forward to seeing your test results!

Justin

Posted in Application-based benchmarks, Collaborative benchmark development, HDXPRT, HDXPRT 2014, Performance benchmarking, Windows 10 |

Creating a machine-learning benchmark

By Bill Catchings

on December 22, 2016

Recently, we wrote about one of the most exciting emerging technology areas, machine learning, and the question of what role the XPRTs could play in the field.

Experts expect machine learning to be the analytics backbone of the IoT data explosion. It is a disruptive technology with potential to influence a broad range of industries. Consumer and industrial applications that take advantage of machine-learning advancements in computer vision, natural language processing, and data analytics are already available and many more are on the way.

Currently, there is no comprehensive machine-learning or deep-learning benchmark that includes home, automotive, industrial, and retail use cases. The challenge with developing a benchmark for machine learning is that these are still the early days of the technology. A fragmented software and hardware landscape and lack of standardized implementations makes benchmarking machine learning complex and challenging.

Based on the conversations we’ve had over the last few weeks, we’ve decided to take on that challenge. With the community’s help, of course!

As we outlined in a blog entry last month, we will work with interested folks in the community, key vendors, and academia to pull together what we are internally calling MLXPRT.

While the result may differ substantially from the existing XPRTs, we think the need for something is great. Whether that will turn out to be a packaged tool or just sample code and workloads remains to be seen.

What we need most your help. We need both general input about what you would like to see as well as any expertise you may have. Let us know any questions you may have or ways you can help.

On a related note, I’ll be at CES 2017 in Las Vegas during the first week of January. I’d love to meet and talk more about machine learning, benchmarking, or the XPRTs. If you’re planning to be there and would like to connect, let us know.

We will not have a blog entry next week over the holidays, so we wish all of you a wonderful time with your families and a great start to the new year.

Bill

Posted in BenchmarkXPRT, BenchmarkXPRT development community, CES, Collaborative benchmark development, Machine learning, What makes a good benchmark? |

Exploring virtual reality

By Eric Hale

on December 15, 2016

We’ve talked a lot in recent weeks about new technologies we are evaluating for the XPRTs. You may remember that back in June, we also wrote about sponsoring a second senior project with North Carolina State University. Last week, the project ended with the traditional Posters and Pies event. The team gave a very well thought‑out presentation.

As you can tell from the photo below, the team designed and implemented a nifty virtual reality app. It’s a room escape puzzle, and it looks great!

The app is a playable game with the ability to record the gameplay for doing repeatable tests. It also includes a recording that allows you to test a device without playing the game. Finally, the app lets you launch directly into the prerecorded game without using a viewer, which will be handy for testing multiple devices.

The team built the app using the Google Cardboard API and the Unity game engine, which allowed them to create Android and iOS versions. We’re looking forward to seeing what that may tell us!

After Posters and Pies, the team came to PT to present their work and answer questions. We were all very impressed with their knowledge and with how well thought out the application was.

Many thanks to team members Christian McCurdy, Gregory Manning, Grayson Jones, and Shon Ferguson (not shown).

Thanks also to Dr. Lina Battestilli, the team’s technical advisor, and Margaret Heil, Director of the Senior Design Center.

We are currently evaluating the app, and expect to make it available to the community in early 2017!

Eric

Posted in Android, Benchmarking, Collaborative benchmark development, iOS, North Carolina State University, Virtual reality |

BatteryXPRT’s future

By Justin Greene

on December 1, 2016

A few weeks ago, we discussed the future of HDXPRT. This week, we’re focusing on the current state of BatteryXPRT 2014 for Android, and how the benchmark may evolve in 2017.

BatteryXPRT continues to provide users with reliable evaluations of their Android device’s performance and battery life under real-world conditions. Originally designed to be compatible with Android 4.2 (Jelly Bean) and above, the benchmark continues to work well on subsequent versions of Android, up to and including Android 6.0 (Marshmallow).

Since Android 7 (Nougat) began to roll out on select devices in the past few months, our internal testing has shown that we’ll need to adjust the BatteryXPRT source code to maintain compatibility with devices running Android 7 and above. We developed the existing source when Eclipse was the officially supported SDK environment, and now we need to bring the code in line with the current Android Studio SDK.

Practically speaking, BatteryXPRT does run on Nougat, and to the best of our knowledge, battery life results are still accurate and reliable. However, the test will not produce a performance score. As more Nougat devices are released in the coming months, it’s possible that other aspects of the benchmark may encounter issues. If this happens during your testing, we encourage you to let us know.

Because MobileXPRT 2015 and BatteryXPRT 2014 performance workloads are so closely related, the next obvious question is whether MobileXPRT 2015 runs on Nougat devices. As of now, MobileXPRT 2015 does run successfully and reliably on Android 7, and this is because the most recent build of MobileXPRT 2015 was compiled using a newer SDK.

We think the best course of action for MobileXPRT 2015 and BatteryXPRT will be to eventually combine them into a single, easy-to-use Android benchmark for performance and battery life. We’ll talk more about that plan in the coming months, and we look forward to hearing your input. Until that transition is successful, we’ll continue to support both BatteryXPRT and MobileXPRT 2015.

As always, we welcome your feedback on these developments, as well as any ideas you may have for future XPRTs.

Justin

Posted in Android, Android L, Android M, Android N, Battery life, BatteryXPRT 2014 for Android, Collaborative benchmark development, MobileXPRT 2015 |

Machine learning

By Bill Catchings

on November 23, 2016

A couple months ago I wrote about doing an inventory of our XPRT tools. Part of that is taking a close look at the six existing XPRTs. The first result of that effort was what I recently wrote about HDXPRT. We’re also looking at emerging technology areas where the BenchmarkXPRT Community has expertise that can guide us.

One of the most exciting of these areas is machine learning. It has rapidly gone from interesting theoretical research (they called them “neural nets” back when I was getting my computer science degree) to something we all use whether we realize it or not. Machine learning (or deep learning) is in everything from intelligent home assistants to autonomous automobiles to industrial device monitoring to personalized shopping in retail environments.

The challenge with developing a benchmark for machine learning is that these are still the early days of the technology. In the past, XPRTs have targeted technologies later in the product cycle. We’re wondering how the XPRT model and the members of its community can play a role here.

One possible use of a machine-learning XPRT is with drones, a market that includes many vendors. Consumers, hobbyists, builders, and the companies creating off-the-shelf models could all benefit from tools and techniques that fairly compare drone performance.

The best approach we’ve come up with to define a machine-learning XPRT starts with identifying common areas such as computer vision, natural language processing, and data analytics, and then, within each of those areas, identifying common algorithms such as AlexNet, GoogLeNet, and VGG. We would also look at the commonly used frameworks such as Caffe, Theano, TensorFlow, and CNTK.

The result might differ from an existing XPRT where you simply run a tool and get a result. Instead, it might take the form of sample code and workloads. Or, maybe even one or two executables that could be used in the most common environments.

At this point, our biggest question is, What do you think? Is this an area you’re interested in? If so, what would you like to see a machine-learning XPRT do?

We’re actively engaging with people in these emerging markets to gauge their interest as well. Regardless of the feedback, we’re excited about the possibilities!