BenchmarkXPRT Blog banner

Search Results for: results viewer

Putting together a good WebXPRT workload proposal

Recently, we announced that we’re moving forward with the development of a new AI-focused WebXPRT 4 workload. It will be an auxiliary workload, which means that it will run as a separate, optional test, and it won’t affect existing WebXPRT 4 tests or scores. Although the inspiration for this new workload came from internal WebXPRT discussions—and, let’s face it, from the huge increase in importance of AI—we wanted to remind you that we’re always open to hearing your WebXPRT workload ideas. If you’d like to submit proposals for new workloads, you don’t have to follow a formal process. Just contact us, and we’ll start the conversation.

If you do decide to send us a workload proposal, it will be helpful to know the types of parameters that we keep in mind. Below, we discuss some of the key questions we ask when we evaluate new WebXPRT workload ideas.

Will it be relevant and interesting to real users, lab testers, and tech reviewers?

When considering a WebXPRT workload proposal, the first two criteria are simple: is it relevant in real life, and are people interested in the workload? We created WebXPRT to evaluate device performance using web-based tasks that consumers are likely to experience daily, so real-life relevance has always been an essential requirement for us throughout development. There are many technologies, functions, and use cases that we could test in a web environment, but only some are relevant to common applications or usage patterns and are likely to draw the interest of real users, lab testers, and technical reviewers.

Will it have cross-platform support?

Currently, WebXPRT runs on almost any web browser and almost every device that supports a web browser. We would like to keep that level of cross-platform support when we introduce new workloads. However, technical differences in how various browsers execute tasks make it challenging to include certain scenarios without undermining our cross-platform ideal. When considering any workload proposal, one of the first questions we ask is, “Will it work on all the major browsers and operating systems?”

There are special exceptions to this guideline. For instance, we’re still in the early days of browser-based AI, and it’s unlikely that a new browser-based AI workload will run on every major browser. If it’s a particularly compelling idea, such as the AI scenario we’re currently working on, we may consider including it as an auxiliary test.

Will it differentiate performance between different types of devices?

XPRT benchmarks provide users with accurate measures for evaluating how well target systems or technologies perform specific tasks. With a broadly targeted benchmark like WebXPRT, if the workloads are so heavy that most devices can’t handle them or so light that most devices complete them without being taxed, the results will be of little use for helping buyers evaluating systems and making purchasing decisions, OEM labs, and the tech press.

That’s why, with any new WebXPRT workload, we look for a sweet spot with respect to how computationally demanding it will be. We want it to run on a wide range of devices—from low-end devices that are several years old to brand-new high-end devices, and everything in between. We also want users to see a wide range of workload scores and resulting overall scores that accurately reflect the experiences those systems deliver, so they can easily grasp the different performance capabilities of the devices under test.

Will results be consistent and easily replicated?

Finally, WebXPRT workloads should produce scores that consistently fall within an acceptable margin of error and are easily replicated with additional testing or comparable gear. Some web technologies are very sensitive to uncontrollable or unpredictable variables, such as internet speed. A workload that measures one of those technologies would be unlikely to produce results that are consistent and easily replicated.

We hope this post will be useful if you’re thinking about potential new workloads that you’d like to see in WebXPRT. If you have any general thoughts about browser performance testing or specific workload ideas that you’d like us to consider, please let us know.

Justin

Making progress with WebXPRT 4 in iOS 17

In recent blog posts, we discussed an issue that we encountered when attempting to run WebXPRT 4 on iOS 17 devices. If you missed those posts, you can find more details about the nature of the problem here. In short, the issue is that the Encrypt Notes and OCR scan subtest in WebXPRT 4 gets stuck when the Tesseract.js Optical Character Recognition (OCR) engine attempts to scan a shopping receipt. We’ve verified that the issue occurs on devices running iOS 17, iPadOS 17, and macOS Sonoma with Safari 17.

After a good bit of troubleshooting and research to try and identify the cause of the problem, we decided to build an updated version of WebXPRT 4 that uses a newer version of Tesseract for the OCR task. Aside from updating Tesseract in the new build, we aimed to change as little as possible. To try and maximize continuity, we’re still using the original input image for the receipt scanning task, and we decided to stick with using the WASM library instead of a WASM-SIMD library. Aside from a new version of tesseract.js, WebXPRT 4 version number updates, and updated documentation where necessary, all other aspects of WebXPRT 4 will remain the same.

We’re currently testing a candidate build of this new version on a wide array of devices. The results so far seem promising, but we want to complete our due diligence and make sure this is the best approach to solving the problem. We know that OEM labs and tech reviewers put a lot of time and effort into compiling databases of results, so we hope to provide a solution that minimizes results disruption and inconvenience for WebXPRT 4 users. Ideally, folks would be able to integrate scores from the new build without any questions or confusion about comparability.

We don’t yet have an exact release date for a new WebXPRT 4 build, but we can say that we’re shooting for the end of October. We appreciate everyone’s patience as we work towards the best possible solution. If you have any questions or concerns about an updated version of WebXPRT 4, please let us know.

Justin

How we evaluate new WebXPRT workload proposals

A key value of the BenchmarkXPRT Development Community is our openness to user feedback. Whether it’s positive feedback about our benchmarks, constructive criticism, ideas for completely new benchmarks, or proposed workload scenarios for existing benchmarks, we appreciate your input and give it serious consideration.

We’re currently accepting ideas and suggestions for ways we can improve WebXPRT 4. We are open to adding both non-workload features and new auxiliary tests, which can be experimental or targeted workloads that run separately from the main test and produce their own scores. You can read more about experimental WebXPRT 4 workloads here. However, a recent user question about possible WebGPU workloads has prompted us to explain the types of parameters that we consider when we evaluate a new WebXPRT workload proposal.

Community interest and real-life relevance

The first two parameters we use when evaluating a WebXPRT workload proposal are straightforward: are people interested in the workload and is it relevant to real life? We originally developed WebXPRT to evaluate device performance using the types of web-based tasks that people are likely to encounter daily, and real-life relevancy continues to be an important criterion for us during development. There are many technologies, functions, and use cases that we could test in a web environment, but only some of them are both relevant to common applications or usage patterns and likely to be interesting to lab testers and tech reviewers.

Maximum cross-platform support

Currently, WebXPRT runs in almost any web browser, on almost any device that has a web browser, and we would ideally maintain that broad level of cross-platform support when introducing new workloads. However, technical differences in the ways that different browsers execute tasks mean that some types of scenarios would be impossible to include without breaking our cross-platform commitment.

One reason that we’re considering auxiliary workloads with WebXPRT, e.g., a battery life rundown, is that those workloads would allow WebXPRT to offer additional value to users while maintaining the cross-platform nature of the main test. Even if a battery life test ran on only one major browser, it could still be very useful to many people.

Performance differentiation

Computer benchmarks such as the XPRTs exist to provide users with reliable metrics that they can use to gauge how well target platforms or technologies perform certain tasks. With a broadly targeted benchmark such as WebXPRT, if the workloads are so heavy that most devices can’t handle them, or so light that most devices complete them without being taxed, the results will have little to no use for OEM labs, the tech press, or independent users when evaluating devices or making purchasing decisions.

Consequently, with any new WebXPRT workload, we try to find a sweet spot in terms of how demanding it is. We want it to run on a wide range of devices—from low-end devices that are several years old to brand-new high-end devices and everything in between. We also want users to see a wide range of workload scores and resulting overall scores, so they can easily grasp the different performance capabilities of the devices under test.

Consistency and replicability

Finally, workloads should produce scores that consistently fall within an acceptable margin of error, and are easily to replicate with additional testing or comparable gear. Some web technologies are very sensitive to uncontrollable or unpredictable variables, such as internet speed. A workload that measures one of those technologies would be unlikely to produce results that are consistent and easily replicated.

We hope this post will be useful for folks who are contemplating potential new WebXPRT workloads. If you have any general thoughts about browser performance testing, or specific workload ideas that you’d like us to consider, please let us know.

Justin

Introducing the XPRT Selector

We’re proud of all the XPRT tools, each of which serves a different purpose for the people who rely on them. But for those new to the XPRTs, we wanted a way to make it easy to tell which tool will best suit each person’s specific requirements. To that end, today we’re excited to announce the XPRT Selector, an interactive web tool that helps consumers, developers, manufacturers, and reviewers zero in on exactly which XPRT tool is the right match for their needs.

Using the XPRT Selector is easy. Simply spin the dials on the wheel to choose the categories that best describe yourself, the devices and operating systems you’re working with, and the topic that interests you. Once you’ve aligned the dials, click Get results, and the Selector will present all the free XPRT tools and resources that are available to you. Along with choosing the best tools for you, the XPRT Selector also explains the purpose and capabilities of each tool.

To see the Selector in action, check out the short video below. You can take the XPRT Selector for a spin at http://www.principledtechnologies.com/benchmarkxprt/the-xprt-selector/.

The XPRT Selector

All the XPRT tools have one thing in common: They help take the guesswork out of device evaluation and comparison, making them invaluable for anyone using, making, or writing about tech products. We think the XPRT Selector is a great addition to the fold!

Justin

XPRTs around the world

This week, we posted an updated version of our “XPRTs around the world” infographic. From time to time, we like to give readers a broader view of the impact that the XPRTs are having around the world, and the infographic shows just how far and wide the XPRTs’ reach has grown.

Here are some numbers from the latest update:

  • The XPRTs have been mentioned more than 7,800 times on over 2,500 unique sites.
  • Those mentions include more than 6,800 articles and reviews.
  • Those mentions originated in over 400 cities located in 58 countries on six continents. If you’re a tech reviewer based in Antarctica, we’re counting on you to help us make it a clean sweep!
  • The BenchmarkXPRT Development Community now includes 203 members from 73 companies and organizations around the world.


In addition to the reach numbers, we’re excited that the XPRTs have now delivered more than 250,000 real-world results!

If you’re familiar with the run counter on WebXPRT.com, you may have noticed that the WebXPRT run tally is rising quickly. Starting with the original release of WebXPRT in early 2013, it took more than three and a half years for the combined run tally of WebXPRT 2013 and WebXPRT 2015 to reach 100,000. In the nine months since that happened, users have added 60,000 runs. The pace is picking up significantly!

We’re grateful for everyone who’s helped us get this far. Here’s to another quarter-million runs and downloads!

Justin

Refining

You may have noticed that we’ve been making some changes to the BenchmarkXPRT pages. We’ve posted links to recent results on the main BenchmarkXPRT page, added a “What’s hot” box highlighting the most recent postings, made it easier to use the benchmark viewer to navigate across benchmarks, and more. We’ll also be including links to the latest news at the end of the blog each week.

The goal of these changes is to make it easier for you to find the information you want, with particular emphasis on making the most recent events easy to find. This is an ongoing process, and we expect to be making additional incremental changes to the pages in the weeks and months ahead.

If you’re a regular user of the BenchmarkXPRT pages, we’d love to hear from you! Are there things you have trouble finding? Do you have any ideas about how to improve the navigation? Is there anything you think works particularly well and do not want to change? All ideas, compliments, and complaints are welcome.

Eric

Last week in the XPRTs

We released MobileXPRT 2015
We added three new TouchXPRT results
We added three new HDXPRT results

Check out the other XPRTs:

Forgot your password?