Category: AI

Thinking ahead at CES 2018

on January 11, 2018

It may sound trite to say that a show like CES is all about the future, but this year’s show is prompting me to think about how our lives will evolve in the coming years. Some technological breakthroughs change the way we do everyday things like play music or hail a cab—while some transform the way we do everything. For technological innovation to truly shift society on a wide scale, it has to coincide with markets of scale in a way that makes life-changing tech accessible to almost everyone (think: smartphones in 2005 versus smartphones in 2018).

These technical and economic forces are coinciding once again in the areas of AI, automation, the Internet of Things (IoT), and consumer robotics. While many of our daily activities will stay the same, the ways we organize and engage with those activities are changing dramatically.

I’ll leave you with a few general observations from the show:

Huawei has a huge presence here. A new tagline for them that I haven’t seen before is a play on their name, “Wow Way.” I suspect we may have a Mate 10 Pro XPRT Spotlight entry in the near future.
The Kino-mo Hypervsn 3D Holograph display blew me away. People were crowding in to see it and couldn’t stop staring. It’s straight out of sci-fi, and its appearance is similar to Princess Leia’s hologram message in Star Wars. (By the way, it looks way better in real life than in the video.)
Sony is making a big push into smart homes by building systems that work cross-platform with a range of smart speakers and assistants. Between their smart home push and some of the cool home theater tech they had on display, I can see them gaining some brand power.
To me, the most exciting concepts at the show involved smart infrastructure, which promises enormous potential to boost the efficient distribution of water, energy, and transportation resources.
Surprisingly, I saw automation, smart city, and smart infrastructure displays from companies that I don’t always associate with IoT or AI, like Bosch and Panasonic. Panasonic was marketing an array of semi-autonomous vehicle cockpit prototypes, and had a section highlighting their partnership with Tesla.
By far, the strangest thing I’ve seen at CES has been the Psychasec booth, staffed by eerie attendants in pure white outfits who talked confidently about “downloading your cortical stack into customized bodies made from organic materials.” The Psychasec staff absolutely refused to break character, which made the whole scene even stranger. Check out the link above for the story behind Psychasec.

And, while I’m probably not supposed to admit this, my favorite part of the show so far has been the line of expensive massage chair vendors doling out free sessions…

More to follow soon,

Justin

Posted in AI, Automated cars, Automation, CES, Holograph, Internet of things, IoT, Trade Shows |

News about WebXPRT and BatteryXPRT

By Justin Greene

on December 7, 2017

Last month, we gave readers a glimpse of the updates in store for the next WebXPRT, and now we have more news to report on that front.

The new version of WebXPRT will be called WebXPRT 3. WebXPRT 3 will retain the convenient features that made WebXPRT 2013 and WebXPRT 2015 our most popular tools, with more than 200,000 combined runs to date. We’ve added new elements, including AI, to a few of the workloads, but the test will still run in 15 minutes or less in most browsers and produce the same easy-to-understand results that help compare browsing performance across a wide variety of devices.

We’re also very close to publishing the WebXPRT 3 Community Preview. For those unfamiliar with our open development community model, BenchmarkXPRT Development Community members have the ability to preview and test new benchmark tools before we release them to the general public. Community previews are a great way for members to evaluate new XPRTs and send us feedback. If you’re interested in joining, you can register here.

In BatteryXPRT news, we recently started to see unusual battery life estimates and high variance when running battery life tests at the default length of 5.25 hours. We think this may be due to changes in how new OS versions are reporting battery life on certain devices, but we’re in the process of extensive testing to learn more. In the meantime, we recommend that BatteryXPRT users adjust the test run time to allow for a full rundown.

Do you have questions or comments about WebXPRT or BatteryXPRT? Let us know!

Justin

Posted in AI, Battery life, BatteryXPRT 2014 for Android, BenchmarkXPRT, BenchmarkXPRT development community, Browser-based benchmarks, Community Preview, Cross-platform benchmarks, Future of performance evaluation, Performance benchmarking, WebXPRT, WebXPRT 3 |

Machine learning performance tool update

By Bill Catchings

on October 19, 2017

Earlier this year we started talking about our efforts to develop a tool to help in evaluating machine learning performance. We’ve given some updates since then, but we’ve also gotten some questions, so I thought I’d do my best to summarize our answers for everyone.

Some have asked what kinds of algorithms we’ve been looking into. As we said in an earlier blog, we’re looking at algorithms involved in computer vision, natural language processing, and data analytics, particularly different aspects of computer vision.

One seemingly trivial question we’ve received regards the proposed name, MLXPRT. We have been thinking of this tool as evaluating machine learning performance, but folks have raised a valid concern that it may well be broader than that. Does machine learning include deep learning? What about other artificial intelligence approaches? I’ve certainly seen other approaches lumped into machine learning, probably because machine learning is the hot topic of the moment. It feels like everything is boasting, “Now with machine learning!”

While there is some value in being part of such a hot movement, we’ve begun to wonder if a more inclusive name, such as AIXPRT, would be better. We’d love to hear your thoughts on that.

We’ve also had questions about the kind of devices the tool will run on. The short answer is that we’re concentrating on edge devices. While there is a need for server AI/ML tools, we’ve been focusing on the evaluating the devices close to the end users. As a result, we’re looking at the inference aspect of machine learning rather than the training aspect.

Probably the most frequent thing we’ve been asked about is the timetable. While we’d hoped to have something available this year, we were overly optimistic. We’re currently working on a more detailed proposal of what the tool will be, and we aim to make that available by the end of this year. If we achieve that goal, our next one will be to have a preliminary version of the tool itself ready in the first half of 2018.

As always, we seek input from folks, like yourself, who are working in these areas. What would you most like to see in an AI/machine learning performance tool? Do you have any questions?

Bill

Posted in AI, Benchmark metrics, Collaborative benchmark development, computer vision, Future of performance evaluation, Machine learning, What makes a good benchmark? |

What’s next for HDXPRT?

By Justin Greene

on October 12, 2017

A few months ago, we discussed some initial ideas for the next version of HDXPRT, including updating the benchmark’s workloads and real-world trial applications and improving the look and feel of the UI. This week, we’d like to share more about the status of the HDXPRT development process.

We’re planning to keep HDXPRT’s three test categories: editing photos, editing music, and converting videos. We’re also planning to use the latest trial versions of the same five applications included in HDXPRT 2014: Adobe Photoshop Elements, Apple iTunes, Audacity, CyberLink MediaEspresso, and HandBrake. The new versions of each of these programs include features and capabilities that may enhance the HDXPRT workloads. For example, Adobe Photoshop Elements 2018 includes interesting new AI tools such as “Open Closed Eyes,” which purports to fix photos ruined by subjects who blinked at the wrong time. We’re evaluating whether any of the new technologies on offer will be a good fit for HDXPRT.

We’re also evaluating how the new Windows 10 SDK and Fall Creators Update will affect HDXPRT. It’s too early to discuss potential changes in any detail, but we know we’ll need to adapt to new development tools, and it’s possible that the Fluent Design System will affect the HDXPRT UI beyond the improvements we already had in mind.

As HDXPRT development progresses, we’ll continue to keep the community up to date. If you have suggestions or insights into the new Fall Creators Update or any of HDXPRT’s real-world applications, we’d love to hear from you! If you’re just reading out about HDXPRT for the first time, you can find out more about the purpose, structure, and capabilities of the test here.

Justin

Posted in AI, Application-based benchmarks, BenchmarkXPRT development community, HDXPRT, HDXPRT development process, HDXPRT release cycle, Performance benchmarking, What makes a good benchmark?, Windows 10, Windows 10 Creators Update |

Thoughts from MWC Shanghai

By Bill Catchings

on June 29, 2017

I’ve spent the last couple days walking the exhibition halls of MWC Shanghai. The Shanghai New International Expo Centre (SNIEC) is large, but smaller than the MWC exhibit space in Barcelona or the set of exhibit halls in Las Vegas for CES. (SNIEC is not even the biggest exhibition space in Shanghai!) Further, MWC here still only took up half the exhibition space, but there was plenty to see. And, I’m less exhausted than after CES or MWC in Barcelona!

If I had to pick one theme from the exhibition halls, it would be 5G. It seemed like half the booths had 5G displayed somewhere in their signage. The cloud was the other concept that seemed to be everywhere. While neither was surprising, it was interesting to see halfway around the world. In truth, it feels like 5G is much farther along here than it is back in the States.

I was also surprised to see how many phone vendors are here that I’d never heard of before such as Lephone and Gionee. I stopped by their booths with XPRT Spotlight information and hope they will send in some of their devices for inclusion in the future.

One thing I found of note was how much technology in general and IoT in particular is going to be everywhere. There was an interesting exhibit showing how stores of the future might operate. I was able to “buy” items without traditionally checking out. (I got a free water and some cookies out of the experience.) I just placed the items in a location on the checkout counter, which read their NFC labels and displayed them on the checkout screen. It seemed sort of like my understanding of the experiments that Amazon has been doing with brick-and-mortar grocery stores (prior to their purchase of Whole Foods). The whole experience felt a bit odd and still unpolished, but I’m sure it will improve and I’ll get used to it.

The next generation will find it not odd, but normal. There were exhibits with groups of children playing with creative technologies from handheld 3D printers to simplified programming languages. They will be the generation after digital natives, maybe the digital creatives? What impact will they have? The future is both exciting and daunting!

I came away from the conference thinking about how the XPRTs can help folks choose amongst the myriad devices and technologies that are just around the corner. What would you most like to see the XPRTs tackle in the next six months to a year?

Bill Catchings

Posted in 5G, AI, computer vision, Education, Future of performance evaluation, Internet of things, IoT, Mobile World Congress, Mobile World Congress, Trade Shows |

Learning about machine learning

By Bill Catchings

on April 27, 2017

Everywhere we look, machine learning is in the news. It’s driving cars and beating the world’s best Go players. Whether we are aware of it or not, it’s in our lives–understanding our voices and identifying our pictures.

Our goal of being able to measure the performance of hardware and software that does machine learning seems more relevant than ever. Our challenge is to scan the vast landscape that is machine learning, and identify which elements to measure first.

There is a natural temptation to see machine learning as being all about neural networks such as AlexNet and GoogLeNet. However, new innovations appear all the time and lots of important work with more classic machine learning techniques is also underway. (Classic machine learning being anything more than a few years old!) Recursive neural networks used for language translation, reinforcement learning used in robotics, and support vector machine (SVM) learning used in text recognition are just a few examples among the wide array of algorithms to consider.

Creating a benchmark or set of benchmarks to cover all those areas, however, is unlikely to be possible. Certainly, creating such an ambitious tool would take so long that it would be of limited usefulness.

Our current thinking is to begin with a small set of representative algorithms. The challenge, of course, is identifying them. That’s where you come in. What would you like to start with?

We anticipate that the benchmark will focus on the types of inference learning and light training that are likely to occur on edge devices. Extensive training with large datasets takes place in data centers or on systems with extraordinary computing capabilities. We’re interested in use cases that will stress the local processing power of everyday devices.

We are, of course, reaching out to folks in the machine learning field—including those in academia, those who create the underlying hardware and software, and those who make the products that rely on that hardware and software.

What do you think?

Bill

Posted in AI, AR, Augmented reality, Automated cars, Automation, computer vision, Future of performance evaluation, Machine learning, What makes a good benchmark? |

Category: AI

Thinking ahead at CES 2018

News about WebXPRT and BatteryXPRT

Machine learning performance tool update

What’s next for HDXPRT?

Thoughts from MWC Shanghai

Learning about machine learning

Check out the other XPRTs: