BenchmarkXPRT Blog banner

Category: Benchmark metrics

AIXPRT: We want your feedback!

Justin Greene
By Justin Greene
on April 12, 2018

Today, we’re publishing the AIXPRT Request for Comments (RFC) document. The RFC explains the need for a new artificial intelligence (AI)/machine learning benchmark, shows how the BenchmarkXPRT Development Community plans to address that need, and provides preliminary design specifications for the benchmark.

We’re seeking feedback and suggestions from anyone interested in shaping the future of machine learning benchmarking, including those not currently part of the Development Community. Usually, only members of the BenchmarkXPRT Development Community have access to our RFCs and the opportunity to provide feedback. However, because we’re seeking input from non-members who have expertise in this field, we will be posting this RFC in the New events & happenings section of the main BenchmarkXPRT.com page and making it available at AIXPRT.com.

We welcome input on all aspects of the benchmark, including scope, workloads, metrics and scores, UI design, and reporting requirements. We will accept feedback through May 13, 2018, after which BenchmarkXPRT Development Community administrators will collect and evaluate the feedback and publish the final design specification.

Please share the RFC with anyone interested in machine learning benchmarking and please send us your feedback before May 13.

Justin

Posted in AI, Benchmark metrics, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, Machine learning, Performance benchmarking, What makes a good benchmark? |

New opportunities for TouchXPRT

Justin Greene
By Justin Greene
on March 29, 2018

Next week’s XPRT Weekly Tech Spotlight will feature a unique device: the HP Envy x2 2-in-1. The first device of its kind on the market, the Envy x2 runs Windows 10 on ARM hardware—in this case, a Qualcomm Snapdragon 835 platform. ASUS and Lenovo will release similar devices in the coming months. Using the ARM chips found in many flagship phones, these devices aim to power robust operating systems on 2-in-1s and laptops while providing extended battery life and always-on LTE connections.

These new devices bring ample opportunities for benchmarking. Consumers will want to know about potential trade-offs between price, power, and battery life—incentivizing tech reviewers to dive into the details and provide useful data points. But for the new Windows on ARM systems, the usual benchmarks have presented challenges. Many traditional laptop benchmarks just won’t work on the new systems. TouchXPRT, however, works like a charm.

TouchXPRT assesses performance on any Windows device. Since it’s a Universal Windows Platform (UWP) app that runs on both x86 and ARM systems, it can evaluate how well a Windows device running on ARM hardware performs compared to traditional laptops and 2-in-1s. It’s easy to install, takes about 15 minutes to run, and you can download it directly from TouchXPRT.com or install it from the Microsoft Store. Labs can also automate testing using the command line or a script.

If you’ve been looking for a Windows performance evaluation tool that’s easy to use and has the flexibility of a UWP app, give TouchXPRT a try. Read more details about TouchXPRT here, and please don’t hesitate to contact us with any questions you may have.

Justin

Posted in Benchmark metrics, Benchmarking, Mobile devices, TouchXPRT, TouchXPRT 2016, TouchXPRT results, Windows 10 |

Principled Technologies and the BenchmarkXPRT Development Community release WebXPRT 3, a free online performance evaluation tool for web-enabled devices

Justin Greene
By Justin Greene
on February 27, 2018

Durham, NC — Principled Technologies and the BenchmarkXPRT Development Community have released WebXPRT 3, a free online tool that gives objective information about how well a laptop, tablet, smartphone, or any other web-enabled device handles common web tasks. Anyone can go to WebXPRT.com and compare existing performance evaluation results on a variety of devices or run a simple evaluation test on their own.

WebXPRT 3 contains six HTML5- and JavaScript-based scenarios created to mirror common web browser tasks: Photo Enhancement, Organize Album Using AI, Stock Option Pricing, Encrypt Notes and OCR Scan, Sales Graphs, and Online Homework.

“WebXPRT is a popular, easy-to-use benchmark run by manufacturers, tech journalists, and consumers all around the world,” said Bill Catchings, co-founder of Principled Technologies, which administers the BenchmarkXPRT Development Community. “We believe that WebXPRT 3 is a great addition to WebXPRT’s legacy of providing relevant and reliable performance data for a wide range of devices.”

WebXPRT is one of the BenchmarkXPRT suite of performance evaluation tools. Other tools include MobileXPRT, TouchXPRT, CrXPRT, BatteryXPRT, and HDXPRT. The XPRTs help users get the facts before they buy, use, or evaluate tech products such as computers, tablets, and phones.

To learn more about and join the BenchmarkXPRT Development Community, go to www.BenchmarkXPRT.com.

About Principled Technologies, Inc.
Principled Technologies, Inc. is a leading provider of technology marketing and learning & development services. It administers the BenchmarkXPRT Development Community.

Principled Technologies, Inc. is located in Durham, North Carolina, USA. For more information, please visit www.PrincipledTechnologies.com.

Company Contact
Justin Greene
BenchmarkXPRT Development Community
Principled Technologies, Inc.
1007 Slater Road, Ste. 300
Durham, NC 27703
BenchmarkXPRTsupport@PrincipledTechnologies.com

Posted in Benchmark metrics, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, Press Releases, WebXPRT, WebXPRT 3, WebXPRT Press Releases | Tagged , , , , |

Looking for performance clues

Eric Hale
By Eric Hale
on October 26, 2017

We’ve written before about how the operating system and other software can influence test scores and even battery life. Benchmarks like the XPRTs provide overall results, but teasing out which factors affect those results may require some detective work. The key is to collect individual data points as evidence to what may be causing performance changes.

The Apple iOS 11 rollout last month is an excellent example of the effect of software on device performance. Angry tweets started almost immediately after the update, claiming that iOS 11 drained device batteries. iPhone users here at PT experienced similar issues. What was the cause of that performance drop? The hardware remained the same. So, did software cause the problem?

Less than a week after the rollout, Mashable published an explanation of possible causes. The article quotes research from mobile security firm Wandera showing that, for the 50,000 “moderate to heavy iPhone and iPad users” in the study, devices running iOS 11 burned through their battery at much faster rates than the same devices running iOS 10. They cite two possible causes:

    • devices often re-categorize the files stored on them for every new OS install, which may account for some of the battery issues.
    • many apps are not optimized for iOS 11 yet.

 

While these explanations make sense, with a little more digging, we could get closer to actually solving the mystery instead of guessing at the causes. After all, it is also possible that people are using iOS 11 differently from iOS 10. So, how could a dedicated sleuth investigate further? Anyone using benchmarks and hands-on testing to sift through various scenarios and configurations could get us closer to solving this mystery and any other software-based performance anomalies. But it’s a daunting task—changing only one variable at a time and recording the results is like pounding the streets and knocking on doors to solve a case.

In all likelihood, some combination of Apple iOS updates and application changes will improve the battery life for iOS 11. In the meantime, we wish we had an XPRT that could test battery life on iOS. Who knows, maybe some future version of WebXPRT will be able to help in future sleuthing.

Eric

Posted in Battery life, Benchmark metrics, Benchmarking, iOS, Uncategorized, WebXPRT |

Machine learning performance tool update

Bill Catchings
By Bill Catchings
on October 19, 2017

Earlier this year we started talking about our efforts to develop a tool to help in evaluating machine learning performance. We’ve given some updates since then, but we’ve also gotten some questions, so I thought I’d do my best to summarize our answers for everyone.

Some have asked what kinds of algorithms we’ve been looking into. As we said in an earlier blog, we’re looking at  algorithms involved in computer vision, natural language processing, and data analytics, particularly different aspects of computer vision.

One seemingly trivial question we’ve received regards the proposed name, MLXPRT. We have been thinking of this tool as evaluating machine learning performance, but folks have raised a valid concern that it may well be broader than that. Does machine learning include deep learning? What about other artificial intelligence approaches? I’ve certainly seen other approaches lumped into machine learning, probably because machine learning is the hot topic of the moment. It feels like everything is boasting, “Now with machine learning!”

While there is some value in being part of such a hot movement, we’ve begun to wonder if a more inclusive name, such as AIXPRT, would be better. We’d love to hear your thoughts on that.

We’ve also had questions about the kind of devices the tool will run on. The short answer is that we’re concentrating on edge devices. While there is a need for server AI/ML tools, we’ve been focusing on the evaluating the devices close to the end users. As a result, we’re looking at the inference aspect of machine learning rather than the training aspect.

Probably the most frequent thing we’ve been asked about is the timetable. While we’d hoped to have something available this year, we were overly optimistic. We’re currently working on a more detailed proposal of what the tool will be, and we aim to make that available by the end of this year. If we achieve that goal, our next one will be to have a preliminary version of the tool itself ready in the first half of 2018.

As always, we seek input from folks, like yourself, who are working in these areas. What would you most like to see in an AI/machine learning performance tool? Do you have any questions?

Bill 

Posted in AI, Benchmark metrics, Collaborative benchmark development, computer vision, Future of performance evaluation, Machine learning, What makes a good benchmark? |

Introducing the WebXPRT 2015 Processor Comparison Chart

Justin Greene
By Justin Greene
on October 5, 2017

Today, we’re excited to announce the WebXPRT 2015 Processor Comparison Chart, a new tool that makes it easier to access hundreds of PT-curated, real-world performance scores from a wide range of devices covering everything from TVs to phones to tablets to PCs.

The chart offers a quick way to browse and compare WebXPRT 2015 results grouped by processor. Unlike benchmark-score charts that may contain results from unknown sources, PT hand-selected each of the results from internal lab testing and reliable tech media sources. If we published multiple scores for an individual processor, the score presented in the chart will be an average of those scores. Users can hover over and click individual score bars for additional information about the test results and test sources for each processor.

WebXPRT proc chart capture

We think the WebXPRT Processor Comparison Chart will be a valuable resource for folks interested in performance testing and product evaluation, but the current iteration is only the beginning. We plan to add additional capabilities on a regular basis, such a detailed filtering and enhanced viewing and navigational options. It’s also possible that we may integrate other XPRT benchmarks down the road.

Most importantly, we want the chart to be a great asset for its users, and that’s where you come in. We’d love to hear your feedback on the features and types of data you’d like to see. If you have suggestions, please let us know!

Justin

Posted in Benchmark metrics, Browser-based benchmarks, Cross-platform benchmarks, Performance benchmarking, Web-based testing, WebXPRT, WebXPRT 2015 |

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2025   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x