BenchmarkXPRT Blog banner

Category: Future of performance evaluation

AIXPRT: We want your feedback!

Today, we’re publishing the AIXPRT Request for Comments (RFC) document. The RFC explains the need for a new artificial intelligence (AI)/machine learning benchmark, shows how the BenchmarkXPRT Development Community plans to address that need, and provides preliminary design specifications for the benchmark.

We’re seeking feedback and suggestions from anyone interested in shaping the future of machine learning benchmarking, including those not currently part of the Development Community. Usually, only members of the BenchmarkXPRT Development Community have access to our RFCs and the opportunity to provide feedback. However, because we’re seeking input from non-members who have expertise in this field, we will be posting this RFC in the New events & happenings section of the main BenchmarkXPRT.com page and making it available at AIXPRT.com.

We welcome input on all aspects of the benchmark, including scope, workloads, metrics and scores, UI design, and reporting requirements. We will accept feedback through May 13, 2018, after which BenchmarkXPRT Development Community administrators will collect and evaluate the feedback and publish the final design specification.

Please share the RFC with anyone interested in machine learning benchmarking and please send us your feedback before May 13.

Justin

MWC18 and technology on the brink

This year’s Mobile World Congress in Barcelona bristled with technologies on the brink of superstardom.  The long-awaited 5G high-speed mobile standard again dominated the conversations, and is one year closer to creating a world of high-speed connections that will make possible mobile usages we’ve only begun to discover.  Intelligent, connected cars promise a self-driving and highly interconnected automotive experience that should ultimately make driving better for all of us.  Artificial intelligence, already a star, showed glimmers of its vast and still barely tapped potential.  In keeping with the show’s name, mobile devices of all sorts proved that phones and tablets and laptops are nowhere near done, with new models and capabilities available all over the many halls that comprised the MWC campus.

Each of those technologies will continue to evolve rapidly over the coming years, and each will create new opportunities for us all to benefit.  Those opportunities will appear both in ways we understand now—faster connections and quicker devices, for example—and in fashions we don’t yet understand.  The new benefits will lead to new usage models, change the ways we interact with the world, and create whole new markets.  (When the first smartphones appeared, they changed photography forever, but that wasn’t their primary goal.)  These new technologies will help us in ways we can now only glimpse.

These changes and new capabilities will breed both competition and, inevitably, confusion.  How are we to know which of the new products deliver the best implementations of these technologies heading toward stardom, and how are we to know when to upgrade to new generations of these products?

Answering those questions, and clarifying some of the confusing aspects of the always shifting tech market, are the reasons the XPRT community and tools exist.  New tech creates new usage models that require new tools to assess–XPRT tools.

If there’s one last lesson I learned from MWC18, it’s that our work is only just beginning.  The new technologies that are on the brink today will become superstars soon, and we’ll be there with the tools you need to assess and compare them.

Mark

WebXPRT 3, Mobile World Congress, and the next HDXPRT

We’re excited about everything that’s in store for the XPRTs, and we want to update community members on what to expect in the next few months.

The next major development is likely to be the WebXPRT 3 general release. We’re currently refining the UI and conducting extensive testing with the community preview build. We’re not ready to announce a firm release date, but hope to do so over the next few weeks. Please try the community preview and give us your feedback, if you haven’t already.

During the last week of February, Mark will be at Mobile World Congress (MWC) in Barcelona. Each year, MWC offers a great opportunity to examine the new trends and technologies that will shape mobile technology in the years to come. We look forward to sharing Mark’s thoughts on this year’s hot topics. Will you be attending MWC this year? If so, let us know!

In addition, we’re hoping to have a community preview of the next HDXPRT ready in the spring. As we mentioned a few months ago, we’re updating the workloads, applications, and UI. For the converting photos scenario, we’re considering incorporating new Adobe Photoshop tools such as the “Open Closed Eyes” feature and an automatic fix for pictures that are out of focus due to handheld camera shake. For the converting videos scenario, we’re including 4K GoPro footage that represents the quality of video captured by today’s “prosumer” demographic.

What features would you like to see in the next HDXPRT? Let us know!

Justin

Machine learning in 2018

We are almost to the end of 2017 and, as you have probably guessed, we will not have a more detailed proposal of our machine learning benchmark ready by the end of the year.

The key aspects of the benchmark proposal we wrote about a few months ago haven’t changed, but we are running behind schedule. We are still hoping to have the proposal ready in Q1 2018 and the tool based on that proposal later in the year. We will keep you posted.

In the meantime, we hope you enjoy as much as we did the recent CGP Grey tech video explanation of machine learning. There are actually two videos—the first one gives a general overview and then the second one does a better job of looking at the current state of machine learning. It talks mainly about the training aspects of machine learning rather than the inference aspects that we are looking into with AIXPRT/MLXPRT.

From all of us in the BenchmarkXPRT Development Community, we hope you and yours have a wonderful holiday and a great start to 2018!

Bill

The WebXPRT 3 Community Preview is here!

Today we’re releasing the WebXPRT 3 Community Preview (CP). As we discussed in the blog last month, in the new version of WebXPRT, we updated the photo-related workloads with new images and a new deep learning task for the Organize Album workload. We also added an optical character recognition task to the Local Notes workload and combined a portion of the DNA Sequence Analysis scenario with a writing sample/spell check scenario to simulate an online homework hub in the new “Online Homework” workload.

Also, longtime WebXPRT users will immediately notice a completely new, but clean and straightforward, UI. We’re still tweaking aspects of the UI and implementing full functionality for certain features such as social media sharing and German language translation, but we don’t anticipate making any significant changes to the overall test or individual workloads before the general release.

As with all community previews, the WebXPRT 3 CP is available only to BenchmarkXPRT Development Community members, who can access the link from the WebXPRT tab in the Members’ Area.

After you try the WebXPRT 3 CP, please send us your comments. Thanks and happy testing!

Justin

News about WebXPRT and BatteryXPRT

Last month, we gave readers a glimpse of the updates in store for the next WebXPRT, and now we have more news to report on that front.

The new version of WebXPRT will be called WebXPRT 3. WebXPRT 3 will retain the convenient features that made WebXPRT 2013 and WebXPRT 2015 our most popular tools, with more than 200,000 combined runs to date. We’ve added new elements, including AI, to a few of the workloads, but the test will still run in 15 minutes or less in most browsers and produce the same easy-to-understand results that help compare browsing performance across a wide variety of devices.

We’re also very close to publishing the WebXPRT 3 Community Preview. For those unfamiliar with our open development community model, BenchmarkXPRT Development Community members have the ability to preview and test new benchmark tools before we release them to the general public. Community previews are a great way for members to evaluate new XPRTs and send us feedback. If you’re interested in joining, you can register here.

In BatteryXPRT news, we recently started to see unusual battery life estimates and high variance when running battery life tests at the default length of 5.25 hours. We think this may be due to changes in how new OS versions are reporting battery life on certain devices, but we’re in the process of extensive testing to learn more. In the meantime, we recommend that BatteryXPRT users adjust the test run time to allow for a full rundown.

Do you have questions or comments about WebXPRT or BatteryXPRT? Let us know!

Justin

Check out the other XPRTs:

Forgot your password?