Category: AI

AIXPRT: We want your feedback!

on April 12, 2018

Today, we’re publishing the AIXPRT Request for Comments (RFC) document. The RFC explains the need for a new artificial intelligence (AI)/machine learning benchmark, shows how the BenchmarkXPRT Development Community plans to address that need, and provides preliminary design specifications for the benchmark.

We’re seeking feedback and suggestions from anyone interested in shaping the future of machine learning benchmarking, including those not currently part of the Development Community. Usually, only members of the BenchmarkXPRT Development Community have access to our RFCs and the opportunity to provide feedback. However, because we’re seeking input from non-members who have expertise in this field, we will be posting this RFC in the New events & happenings section of the main BenchmarkXPRT.com page and making it available at AIXPRT.com.

We welcome input on all aspects of the benchmark, including scope, workloads, metrics and scores, UI design, and reporting requirements. We will accept feedback through May 13, 2018, after which BenchmarkXPRT Development Community administrators will collect and evaluate the feedback and publish the final design specification.

Please share the RFC with anyone interested in machine learning benchmarking and please send us your feedback before May 13.

Justin

Posted in AI, Benchmark metrics, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, Machine learning, Performance benchmarking, What makes a good benchmark? |

MWC18 and technology on the brink

By Mark Van Name

on March 8, 2018

This year’s Mobile World Congress in Barcelona bristled with technologies on the brink of superstardom. The long-awaited 5G high-speed mobile standard again dominated the conversations, and is one year closer to creating a world of high-speed connections that will make possible mobile usages we’ve only begun to discover. Intelligent, connected cars promise a self-driving and highly interconnected automotive experience that should ultimately make driving better for all of us. Artificial intelligence, already a star, showed glimmers of its vast and still barely tapped potential. In keeping with the show’s name, mobile devices of all sorts proved that phones and tablets and laptops are nowhere near done, with new models and capabilities available all over the many halls that comprised the MWC campus.

Each of those technologies will continue to evolve rapidly over the coming years, and each will create new opportunities for us all to benefit. Those opportunities will appear both in ways we understand now—faster connections and quicker devices, for example—and in fashions we don’t yet understand. The new benefits will lead to new usage models, change the ways we interact with the world, and create whole new markets. (When the first smartphones appeared, they changed photography forever, but that wasn’t their primary goal.) These new technologies will help us in ways we can now only glimpse.

These changes and new capabilities will breed both competition and, inevitably, confusion. How are we to know which of the new products deliver the best implementations of these technologies heading toward stardom, and how are we to know when to upgrade to new generations of these products?

Answering those questions, and clarifying some of the confusing aspects of the always shifting tech market, are the reasons the XPRT community and tools exist. New tech creates new usage models that require new tools to assess–XPRT tools.

If there’s one last lesson I learned from MWC18, it’s that our work is only just beginning. The new technologies that are on the brink today will become superstars soon, and we’ll be there with the tools you need to assess and compare them.

Mark

Posted in 5G, AI, Automated cars, BenchmarkXPRT development community, Future of performance evaluation, Mobile World Congress, Mobile World Congress, Trade Shows |

Thinking ahead at CES 2018

By Justin Greene

on January 11, 2018

It may sound trite to say that a show like CES is all about the future, but this year’s show is prompting me to think about how our lives will evolve in the coming years. Some technological breakthroughs change the way we do everyday things like play music or hail a cab—while some transform the way we do everything. For technological innovation to truly shift society on a wide scale, it has to coincide with markets of scale in a way that makes life-changing tech accessible to almost everyone (think: smartphones in 2005 versus smartphones in 2018).

These technical and economic forces are coinciding once again in the areas of AI, automation, the Internet of Things (IoT), and consumer robotics. While many of our daily activities will stay the same, the ways we organize and engage with those activities are changing dramatically.

I’ll leave you with a few general observations from the show:

Huawei has a huge presence here. A new tagline for them that I haven’t seen before is a play on their name, “Wow Way.” I suspect we may have a Mate 10 Pro XPRT Spotlight entry in the near future.
The Kino-mo Hypervsn 3D Holograph display blew me away. People were crowding in to see it and couldn’t stop staring. It’s straight out of sci-fi, and its appearance is similar to Princess Leia’s hologram message in Star Wars. (By the way, it looks way better in real life than in the video.)
Sony is making a big push into smart homes by building systems that work cross-platform with a range of smart speakers and assistants. Between their smart home push and some of the cool home theater tech they had on display, I can see them gaining some brand power.
To me, the most exciting concepts at the show involved smart infrastructure, which promises enormous potential to boost the efficient distribution of water, energy, and transportation resources.
Surprisingly, I saw automation, smart city, and smart infrastructure displays from companies that I don’t always associate with IoT or AI, like Bosch and Panasonic. Panasonic was marketing an array of semi-autonomous vehicle cockpit prototypes, and had a section highlighting their partnership with Tesla.
By far, the strangest thing I’ve seen at CES has been the Psychasec booth, staffed by eerie attendants in pure white outfits who talked confidently about “downloading your cortical stack into customized bodies made from organic materials.” The Psychasec staff absolutely refused to break character, which made the whole scene even stranger. Check out the link above for the story behind Psychasec.

And, while I’m probably not supposed to admit this, my favorite part of the show so far has been the line of expensive massage chair vendors doling out free sessions…

More to follow soon,

Justin

Posted in AI, Automated cars, Automation, CES, Holograph, Internet of things, IoT, Trade Shows |

News about WebXPRT and BatteryXPRT

By Justin Greene

on December 7, 2017

Last month, we gave readers a glimpse of the updates in store for the next WebXPRT, and now we have more news to report on that front.

The new version of WebXPRT will be called WebXPRT 3. WebXPRT 3 will retain the convenient features that made WebXPRT 2013 and WebXPRT 2015 our most popular tools, with more than 200,000 combined runs to date. We’ve added new elements, including AI, to a few of the workloads, but the test will still run in 15 minutes or less in most browsers and produce the same easy-to-understand results that help compare browsing performance across a wide variety of devices.

We’re also very close to publishing the WebXPRT 3 Community Preview. For those unfamiliar with our open development community model, BenchmarkXPRT Development Community members have the ability to preview and test new benchmark tools before we release them to the general public. Community previews are a great way for members to evaluate new XPRTs and send us feedback. If you’re interested in joining, you can register here.

In BatteryXPRT news, we recently started to see unusual battery life estimates and high variance when running battery life tests at the default length of 5.25 hours. We think this may be due to changes in how new OS versions are reporting battery life on certain devices, but we’re in the process of extensive testing to learn more. In the meantime, we recommend that BatteryXPRT users adjust the test run time to allow for a full rundown.

Do you have questions or comments about WebXPRT or BatteryXPRT? Let us know!

Justin

Posted in AI, Battery life, BatteryXPRT 2014 for Android, BenchmarkXPRT, BenchmarkXPRT development community, Browser-based benchmarks, Community Preview, Cross-platform benchmarks, Future of performance evaluation, Performance benchmarking, WebXPRT, WebXPRT 3 |

Machine learning performance tool update

By Bill Catchings

on October 19, 2017

Earlier this year we started talking about our efforts to develop a tool to help in evaluating machine learning performance. We’ve given some updates since then, but we’ve also gotten some questions, so I thought I’d do my best to summarize our answers for everyone.

Some have asked what kinds of algorithms we’ve been looking into. As we said in an earlier blog, we’re looking at algorithms involved in computer vision, natural language processing, and data analytics, particularly different aspects of computer vision.

One seemingly trivial question we’ve received regards the proposed name, MLXPRT. We have been thinking of this tool as evaluating machine learning performance, but folks have raised a valid concern that it may well be broader than that. Does machine learning include deep learning? What about other artificial intelligence approaches? I’ve certainly seen other approaches lumped into machine learning, probably because machine learning is the hot topic of the moment. It feels like everything is boasting, “Now with machine learning!”

While there is some value in being part of such a hot movement, we’ve begun to wonder if a more inclusive name, such as AIXPRT, would be better. We’d love to hear your thoughts on that.

We’ve also had questions about the kind of devices the tool will run on. The short answer is that we’re concentrating on edge devices. While there is a need for server AI/ML tools, we’ve been focusing on the evaluating the devices close to the end users. As a result, we’re looking at the inference aspect of machine learning rather than the training aspect.

Probably the most frequent thing we’ve been asked about is the timetable. While we’d hoped to have something available this year, we were overly optimistic. We’re currently working on a more detailed proposal of what the tool will be, and we aim to make that available by the end of this year. If we achieve that goal, our next one will be to have a preliminary version of the tool itself ready in the first half of 2018.

As always, we seek input from folks, like yourself, who are working in these areas. What would you most like to see in an AI/machine learning performance tool? Do you have any questions?

Bill

Posted in AI, Benchmark metrics, Collaborative benchmark development, computer vision, Future of performance evaluation, Machine learning, What makes a good benchmark? |

What’s next for HDXPRT?

By Justin Greene

on October 12, 2017

A few months ago, we discussed some initial ideas for the next version of HDXPRT, including updating the benchmark’s workloads and real-world trial applications and improving the look and feel of the UI. This week, we’d like to share more about the status of the HDXPRT development process.

We’re planning to keep HDXPRT’s three test categories: editing photos, editing music, and converting videos. We’re also planning to use the latest trial versions of the same five applications included in HDXPRT 2014: Adobe Photoshop Elements, Apple iTunes, Audacity, CyberLink MediaEspresso, and HandBrake. The new versions of each of these programs include features and capabilities that may enhance the HDXPRT workloads. For example, Adobe Photoshop Elements 2018 includes interesting new AI tools such as “Open Closed Eyes,” which purports to fix photos ruined by subjects who blinked at the wrong time. We’re evaluating whether any of the new technologies on offer will be a good fit for HDXPRT.

We’re also evaluating how the new Windows 10 SDK and Fall Creators Update will affect HDXPRT. It’s too early to discuss potential changes in any detail, but we know we’ll need to adapt to new development tools, and it’s possible that the Fluent Design System will affect the HDXPRT UI beyond the improvements we already had in mind.

As HDXPRT development progresses, we’ll continue to keep the community up to date. If you have suggestions or insights into the new Fall Creators Update or any of HDXPRT’s real-world applications, we’d love to hear from you! If you’re just reading out about HDXPRT for the first time, you can find out more about the purpose, structure, and capabilities of the test here.

Justin

Posted in AI, Application-based benchmarks, BenchmarkXPRT development community, HDXPRT, HDXPRT development process, HDXPRT release cycle, Performance benchmarking, What makes a good benchmark?, Windows 10, Windows 10 Creators Update |

Category: AI

AIXPRT: We want your feedback!

MWC18 and technology on the brink

Thinking ahead at CES 2018

News about WebXPRT and BatteryXPRT

Machine learning performance tool update

What’s next for HDXPRT?

Check out the other XPRTs: