Category: Machine learning

Creating a machine-learning benchmark

on December 22, 2016

Recently, we wrote about one of the most exciting emerging technology areas, machine learning, and the question of what role the XPRTs could play in the field.

Experts expect machine learning to be the analytics backbone of the IoT data explosion. It is a disruptive technology with potential to influence a broad range of industries. Consumer and industrial applications that take advantage of machine-learning advancements in computer vision, natural language processing, and data analytics are already available and many more are on the way.

Currently, there is no comprehensive machine-learning or deep-learning benchmark that includes home, automotive, industrial, and retail use cases. The challenge with developing a benchmark for machine learning is that these are still the early days of the technology. A fragmented software and hardware landscape and lack of standardized implementations makes benchmarking machine learning complex and challenging.

Based on the conversations we’ve had over the last few weeks, we’ve decided to take on that challenge. With the community’s help, of course!

As we outlined in a blog entry last month, we will work with interested folks in the community, key vendors, and academia to pull together what we are internally calling MLXPRT.

While the result may differ substantially from the existing XPRTs, we think the need for something is great. Whether that will turn out to be a packaged tool or just sample code and workloads remains to be seen.

What we need most your help. We need both general input about what you would like to see as well as any expertise you may have. Let us know any questions you may have or ways you can help.

On a related note, I’ll be at CES 2017 in Las Vegas during the first week of January. I’d love to meet and talk more about machine learning, benchmarking, or the XPRTs. If you’re planning to be there and would like to connect, let us know.

We will not have a blog entry next week over the holidays, so we wish all of you a wonderful time with your families and a great start to the new year.

Bill

Posted in BenchmarkXPRT, BenchmarkXPRT development community, CES, Collaborative benchmark development, Machine learning, What makes a good benchmark? |

Machine learning

By Bill Catchings

on November 23, 2016

A couple months ago I wrote about doing an inventory of our XPRT tools. Part of that is taking a close look at the six existing XPRTs. The first result of that effort was what I recently wrote about HDXPRT. We’re also looking at emerging technology areas where the BenchmarkXPRT Community has expertise that can guide us.

One of the most exciting of these areas is machine learning. It has rapidly gone from interesting theoretical research (they called them “neural nets” back when I was getting my computer science degree) to something we all use whether we realize it or not. Machine learning (or deep learning) is in everything from intelligent home assistants to autonomous automobiles to industrial device monitoring to personalized shopping in retail environments.

The challenge with developing a benchmark for machine learning is that these are still the early days of the technology. In the past, XPRTs have targeted technologies later in the product cycle. We’re wondering how the XPRT model and the members of its community can play a role here.

One possible use of a machine-learning XPRT is with drones, a market that includes many vendors. Consumers, hobbyists, builders, and the companies creating off-the-shelf models could all benefit from tools and techniques that fairly compare drone performance.

The best approach we’ve come up with to define a machine-learning XPRT starts with identifying common areas such as computer vision, natural language processing, and data analytics, and then, within each of those areas, identifying common algorithms such as AlexNet, GoogLeNet, and VGG. We would also look at the commonly used frameworks such as Caffe, Theano, TensorFlow, and CNTK.

The result might differ from an existing XPRT where you simply run a tool and get a result. Instead, it might take the form of sample code and workloads. Or, maybe even one or two executables that could be used in the most common environments.

At this point, our biggest question is, What do you think? Is this an area you’re interested in? If so, what would you like to see a machine-learning XPRT do?

We’re actively engaging with people in these emerging markets to gauge their interest as well. Regardless of the feedback, we’re excited about the possibilities!

Bill

Posted in Collaborative benchmark development, Drones, Future of performance evaluation, HDXPRT, Machine learning |

Airborne

By Eric Hale

on November 17, 2016

I’m old enough that I’ve never really understood the whole selfie thing. However, it’s clearly never going away, and I’m fascinated–although a little creeped out–by the development of selfie drones. It’s amazing that we have so quickly reached the point where you can buy a drone that will literally fit in your pocket.

As an example of how sophisticated these devices can be, consider Zero Robotics Hover Camera Passport. It’s capable of capturing 4K UHD video and 13-megapixel images, it can track faces or bodies, and it includes sensors, including sonar, to measure the distance from air to ground. All in a package that’s about the size of an old VHS tape.

A while back we talked about the new ways people are finding to use technology, and how the XPRTs need to adapt. While I don’t think we’re going to be seeing DroneXPRT any time soon, we’ve been talking about including the technologies that make these devices possible in the XPRTs. These technologies include machine learning, computer vision, and 4K video.

What new devices fascinate you? Which technologies are going to be most useful in the near future? Let us know!

Eric

Posted in computer vision, Drones, IoT, Machine learning, Mobile devices | Tagged 4K video |

Doing things a little differently

By Eric Hale

on September 8, 2016

I enjoyed watching the Apple Event live yesterday. There were some very impressive announcements. (And a few which were not so impressive – the Breathe app would get on my nerves really fast!)

One thing that I was very impressed by was the ability of the iPhone 7 Plus camera to create depth-of-field effects. Some of the photos demonstrated how the phone used machine learning to identify people in the shot and keep them in focus while blurring the background, creating a shallow depth of field. This causes the subjects in a photo to really stand out. The way we take photos is not the only thing that’s changing. There was a mention of machine learning being part of Apple’s QuickType keyboard, to help with “contextual prediction.”

This is only one product announcement, but it’s a reminder that we need to be constantly examining every part of the XPRTs. Recently, we talked a bit about how people will be using their devices in new ways in the coming months, and we need to be developing tests for these new applications. However, we must also stay focused on keeping existing tests fresh. People will keep taking photos, but today’s photo editing tests may not be relevant a year or two from now.

Were there any announcements yesterday that got you excited? Let us know!

Eric

Posted in Apple, Benchmark metrics, Benchmarking, Future of performance evaluation, Machine learning, Phones, What makes a good benchmark? |

The things we do now

By Eric Hale

on August 18, 2016

We mentioned a couple of weeks ago that the Microsoft Store added an option to indicate holographic support, which we selected for TouchXPRT. So, it was no surprise to see Microsoft announce that next year, they will release an update to Windows 10 that enables mainstream PCs to run the Windows Holographic shell. They also announced that they‘re working with Intel to develop a reference architecture for mixed-reality-ready PCs. Mixed-reality applications, which combine the real world with a virtual reality, demand sophisticated computer vision, and applications that can learn about the world around them.

As we’ve said before, we are constantly watching how people use their devices. One of the most basic principles of the XPRT benchmarks is to test devices using the same kinds of work that people do in the real world. As people find new ways to use their devices, the workloads in the benchmarks should evolve as well. Virtual reality, computer vision, and machine learning are among the technologies we are looking at.

What sorts of things are you doing today that you weren’t a year ago? (Other than Pokémon GO – we know about that one.) Would you like to see those sorts of workloads in the XPRTs? Let us know!

Eric

Posted in Benchmarking, computer vision, Future of performance evaluation, Holograph, Machine learning, TouchXPRT, TouchXPRT 2016, Virtual reality |

Category: Machine learning

Creating a machine-learning benchmark

Machine learning

Airborne

Doing things a little differently

The things we do now

Check out the other XPRTs: