BenchmarkXPRT Blog banner

Category: Microsoft Edge

Web APIs: Possible paths for the AI-focused WebXPRT 4 auxiliary workload

Justin Greene
By Justin Greene
on September 19, 2024

In our last blog post, we discussed one of the major decision points we’re facing as we work on what we hope will be the first new AI-focused WebXPRT 4 auxiliary workload: choosing a Web AI framework. In today’s blog, we’re discussing another significant decision that we need to make for the future workload’s development path: choosing a web API.

Many of you are familiar with the concept of an application programming interface (API). Simply put, APIs implement sets of software rules, tools, and/or protocols that serve as intermediaries that make it possible for different computer programs or components to communicate with each other. APIs simplify many development tasks for programmers and provide standardized ways for applications to share data, functions, and system resources.

Web APIs fulfill the intermediary role of an API—through HTTP-based communication—for web servers (on the server side) or web browsers (on the client side). Client-side web APIs make it possible for browser-based applications to expand browser functionality. They execute the kinds of JavaScript, HTML5, and WebAssembly (Wasm) workloads—among other examples—that support the wide variety of browser extensions many of us use every day. WebXPRT uses those types of browser-based workloads to evaluate system performance. To lay a solid foundation for the first future browser-based AI workload, we need to choose a web API that will be compatible with WebXPRT and the Web AI framework and AI inference workload(s) we ultimately choose.

Currently, there are three main web API paths for running AI inference in a web browser: Web Neural Network (WebNN), Wasm, and WebGPU. These three web technologies are in various stages of development and standardization. Each has different levels of support within the major browsers. Here are basic overviews of each of the three options, as well as a few of our thoughts on the benefits and limitations that each may bring to the table for a future WebXPRT AI workload:

  • WebNN is a JavaScript API that enables developers to directly execute machine learning (ML) tasks on neural networks within web-based applications. WebNN makes it easier to integrate ML models into web apps, and it allows web apps to leverage the power of neural processing units (NPUs). WebNN has a lot going for it. It’s hardware-agnostic and works with various ML frameworks. It’s likely to be a major player in future browser-based inference applications. However, as a web standard, WebNN is still in the development stage and is only available in developer previews for Chromium-based browsers. Full default WebNN support could take a year or more.
  • Wasm is a binary instruction format that works across all modern browsers. Wasm provides a sandboxed environment that operates at near-native speeds and takes advantage of common hardware specs across platforms. Wasm’s capabilities offer web developers a great deal of flexibility for running complex client applications in the browser. Simply put, Wasm can help developers adapt their existing code for additional platforms and browser-based applications without requiring extensive code rewrites. Wasm’s flexibility and cross-platform compatibility is one of the reasons that we’ve already made use of Wasm in two existing WebXPRT 4 workloads that feature AI tasks: Organize Album using AI, and Encrypt Notes and OCR Scan. Wasm can also work together with other web APIs, such as WebGPU.
  • WebGPU enables web-based applications to directly access the graphics rendering and computational capabilities of a system’s GPU. The parallel computational abilities of GPUs make them especially well-suited to efficiently handle some of the demands of AI inference workloads, including image-based GenAI workloads or large language models. Google Chrome and Microsoft Edge currently support WebGPU, and it’s available in Safari through a tech preview.

Right now, we don’t think that WebNN will be fully out of the development phase in time to serve as our go-to web API for a new WebXPRT AI workload. Wasm and/or WebGPU appear to our best options for now. When WebNN is fully baked and available in mainstream browsers, it’s possible that we could port any existing Wasm- or WebGPU-based WebXPRT AI workloads to WebNN, which may open the possibility of cross-platform browser-based NPU performance comparisons.

All that said and as we mentioned in our previous post about Web AI frameworks, we have not made any final decisions about a web API or any aspect of the future workload. We’re still in the early stages of this project. We want your input.

If this discussion has sparked web AI ideas that you think would benefit the process, or if you have feedback you’d like to share, please feel free to contact us!

Justin

Posted in AI, benchmark, Benchmarking, BenchmarkXPRT, browser performance, Browser-based benchmarks, Chrome, Chromium, Collaborative benchmark development, Cross-platform benchmarks, Future of performance evaluation, GenAI, Google, Google Chrome, GPU, HTML5, inference, JavaScript, large language models, Machine learning, Microsoft, Microsoft Edge, NPU, Performance benchmarking, Wasm, Web AI, web API, WebAssembly, WebXPRT, WebXPRT 4 | Tagged , , , , , , , , , , , , , , , , , , |

Updating our WebXPRT 4 browser performance comparisons with new gear

Justin Greene
By Justin Greene
on June 27, 2024

Once or twice per year, we refresh an ongoing series of WebXPRT comparison tests to see if recent updates have reordered the performance rankings of popular web browsers. We published our most recent comparison in January, when we used WebXPRT 4 to compare the performance of five browsers on the same system.

This time, we’re publishing an updated set of comparison scores sooner than we normally would because we chose to move our testing to a newer reference laptop. The previous system—a Dell XPS 13 7930 with an Intel Core i3-10110U processor and 4 GB of RAM—is now several years old. We wanted to transition to a system that is more in line with current mid-range laptops. By choosing to test on a capable mid-tier laptop, our comparison scores are more likely to fall within the range of scores we would see from an typical user today.

Our new reference system is a Lenovo ThinkPad T14s Gen 3 with an Intel Core i7-1270P processor and 16 GB of RAM. It’s running Windows 11 Home, updated to version 23H2 (22631.3527). Before testing, we installed all current Windows updates and tested on a clean system image. After the update process was complete, we turned off updates to prevent any further updates from interfering with test runs. We ran WebXPRT 4 three times each on five browsers: Brave, Google Chrome, Microsoft Edge, Mozilla Firefox, and Opera. In Figure 1 below, each browser’s score is the median of the three test runs.

In our last round of tests—on the Dell XPS 13—the four Chromium-based browsers (Brave, Chrome, Edge, and Opera) produced close scores, with Edge taking a small lead among the four. Each of the Chromium browsers significantly outperformed Firefox, with the slowest of the Chromium browsers (Brave) outperforming Firefox by 13.5 percent.

In this round of tests—on the Lenovo ThinkPad T14s—the scores were very tight, with a difference of only 4 percent between the last-place browser (Brave) and the winner (Chrome). Interestingly, Firefox no longer trailed the four Chromium browsers—it was squarely in the middle of the pack.

Figure 1: The median scores from running WebXPRT 4 three times with each browser on the Lenovo ThinkPad T14s.

Unlike previous rounds that showed a higher degree of performance differentiation between the browsers, scores from this round of tests are close enough that most users wouldn’t notice a difference. Even if the difference between the highest and lowest scores was substantial, the quality of your browsing experience will often depend on factors such as the types of things you do on the web (e.g., gaming, media consumption, or multi-tab browsing), the impact of extensions on performance, and how frequently the browsers issue updates and integrate new technologies, among other things. It’s important to keep such variables in mind when thinking about how browser performance comparison results may translate to your everyday web experience.

Have you tried using WebXPRT 4 to test the speed of different browsers on the same system? If so, we’d love for you to tell us about it! Also, please tell us what other WebXPRT data you’d like to see!

Justin

Posted in benchmark, Benchmark metrics, Brave, browser performance, Browser-based benchmarks, Chrome, Chromium, Cross-platform benchmarks, Dell, Firefox, Google Chrome, Lenovo, Microsoft Edge, Opera, Performance benchmarking, WebXPRT, WebXPRT 4, Windows 11 | Tagged , , , , , , , , , , , |

Up next for WebXPRT 4: A new AI-focused workload!

Justin Greene
By Justin Greene
on May 30, 2024

We’re always thinking about ways to improve WebXPRT. In the past, we’ve discussed the potential benefits of auxiliary workloads and the role that such workloads might play in future WebXPRT updates and versions. Today, we’re very excited to announce that we’ve decided to move forward with the development of a new WebXPRT 4 workload focused on browser-side AI technology!

WebXPRT 4 already includes timed AI tasks in two of its workloads: the Organize Album using AI workload and the Encrypt Notes and OCR Scan workload. These two workloads reflect the types of light browser-side inference tasks that have been available for a while now, but most heavy-duty inference on the web has historically happened in on-prem servers or in the cloud. Now, localized AI technology is growing by leaps and bounds, and the integration of new AI capabilities with browser-based tasks is on the threshold of advancing rapidly.

Because of this growth, we believe now is the time to start work on giving WebXPRT 4 the ability to evaluate new browser-based AI capabilities—capabilities that are likely to become a part of everyday life in the next few years. We haven’t yet decided on a test scenario or software stack for the new workload, but we’ll be working to refine our plan in the coming months. There seems to be some initial promise in emerging frameworks such as ONNX Runtime Web, which allows users to run and deploy web-based machine learning models by using JavaScript APIs and libraries. In addition, new Web APIs like WebGPU (currently supported in Edge, Chrome, and tech preview in Safari) and WebNN (in development) may soon help facilitate new browser-side AI workloads.

We know that many longtime WebXPRT 4 users will have questions about how this new workload may affect their tests. We want to assure you that the workload will be an optional bonus workload and will not run by default during normal WebXPRT 4 tests. As you consider possibilities for the new workload, here are a few points to keep in mind:

  • The workload will be optional for users to run.
  • It will not affect the main WebXPRT 4 subtest or overall scores in any way.
  • It will run separately from the main test and will produce its own score(s).
  • Current and future WebXPRT 4 results will still be comparable to one another, so users who’ve already built a database of WebXPRT 4 scores will not have to retest their devices.
  • Because many of the available frameworks don’t currently run on all browsers, the workload may not run on every platform.

As we research available technologies and explore our options, we would love to hear from you. If you have ideas for an AI workload scenario that you think would be useful or thoughts on how we should implement it, please let us know! We’re excited about adding new technologies and new value to WebXPRT 4, and we look forward to sharing more information here in the blog as we make progress.

Justin

Posted in AI, benchmark, BenchmarkXPRT, BenchmarkXPRT development community, browser performance, Browser-based benchmarks, Chrome, Collaborative benchmark development, Future of performance evaluation, JavaScript, Microsoft Edge, on-device AI, ONNX Runtime Web, Performance benchmarking, Safari, WebGPU, WebNN, WebXPRT, WebXPRT 4 | Tagged , , , , , , , , , , , |

Comparing the WebXPRT 4 performance of five popular browsers

Justin Greene
By Justin Greene
on January 25, 2024

Every so often, we like to refresh a series of in-house WebXPRT comparison tests to see if recent updates have changed the performance rankings of popular web browsers. We published our most recent comparison last February, when we used WebXPRT 4 to compare the performance of five browsers on the same system.

For this round of tests, we used the same Dell XPS 13 7930 laptop as last time, which features an Intel Core i3-10110U processor and 4 GB of RAM, running Windows 11 Home updated to version 23H2 (22631.307). We installed all current Windows updates, and updated each of the browsers under test: Brave, Google Chrome, Microsoft Edge, Mozilla Firefox, and Opera.

After the update process completed, we turned off updates to prevent them from interfering with test runs. We ran WebXPRT 4 three times on each of the five browsers. The score we post for each browser is the median of the three test runs.

In our last round of tests, the range between high and low scores was tight, with an overall difference of only 4.3 percent. Edge squeaked out a win, with a 2.1 percent performance advantage over Chrome. Firefox came in last, but was only one overall score point behind the tied score of Brave and Opera.

In this round of testing, the rank order did not change, but we saw more differentiation in the range of scores. While the performance of each browser improved, the range between high and low scores widened to a 19.1 percent difference between first-place Edge and last-place Firefox. The scores of the four Chromium-based browsers (Brave, Opera, Chrome, and Edge) all improved by at least 21 points, while the Firefox score only improved by one point. 

Do these results mean that Microsoft Edge will always provide a faster web experience, or Firefox will always be slower than the others? Not necessarily. It’s true that a device with a higher WebXPRT score will probably feel faster during daily web activities than one with a much lower score, but your experience depends in part on the types of things you do on the web, along with your system’s privacy settings, memory load, ecosystem integration, extension activity, and web app capabilities.

In addition, browser speed can noticeably increase or decrease after an update, and OS-specific optimizations can affect performance, such as with Edge on Windows 11 and Chrome on Chrome OS. All these variables are important to keep in mind when considering how WebXPRT results may translate to your everyday experience.

Have you used WebXPRT 4 to compare browser performance on the same system? Let us know how it turned out!

Justin

Posted in benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, Brave, browser performance, Browser-based benchmarks, Chromium, Firefox, Google Chrome, Microsoft Edge, Mozilla, Opera, Performance benchmarking, Web-based testing, WebXPRT, WebXPRT 4, Windows 11, Windows Update | Tagged , , , , , , , , , , |

Comparing the performance of popular browsers with WebXPRT 4

Justin Greene
By Justin Greene
on February 2, 2023

If you’ve been reading the XPRT blog for a while, you know that we occasionally like to revisit a series of in-house WebXPRT comparison tests to see if recent updates have changed the performance rankings of popular web browsers. We published our most recent comparison last April, when we used WebXPRT 4 to compare the performance of five browsers on the same system.

For this round of tests, we used a Dell XPS 13 7930, which features an Intel Core i3-10110U processor and 4 GB of RAM, running Windows 11 Home updated to version 22H2 (22621.1105). We installed all current Windows updates, and updated each of the browsers under test: Brave, Google Chrome, Microsoft Edge, Mozilla Firefox, and Opera.

After the update process completed, we turned off updates to prevent them from interfering with test runs. We ran WebXPRT 4 three times on each of the five browsers. The score we post for each browser is the median of the three test runs.

In our last round of tests, Edge was the clear winner, with a 2.2 percent performance advantage over Chrome. Firefox came in last, about 3 percent slower than Opera, which was in the middle of the pack. With updated versions of the browsers, the only change in rank order was that Brave moved into a tie with Opera.

While the rank order from this round of tests was very similar to the previous round, we did observe two clear performance trends: (1) the range between high and low scores was tighter, dropping from a difference of 7.8 percent to 4.3 percent, and (2) every browser demonstrated improved performance. The chart below illustrates both trends. Firefox showed the single largest score improvement at 7.8 percent, but the performance jump for each browser was considerable.

Do these results mean that Microsoft Edge will always provide a speedier web experience, or Firefox will always be slower than the others? Not necessarily. It’s true that a device with a higher WebXPRT score will probably feel faster during daily web activities than one with a much lower score, but your experience depends in part on the types of things you do on the web, along with your system’s privacy settings, memory load, ecosystem integration, extension activity, and web app capabilities.

In addition, browser speed can noticeably increase or decrease after an update, and OS-specific optimizations can affect performance, such as with Edge on Windows 11 and Chrome on Chrome OS. All these variables are important to keep in mind when considering how WebXPRT results translate to your everyday experience.

Have you used WebXPRT to compare browser performance on the same system? Let us know how it turned out!

Justin

Posted in benchmark, Benchmark metrics, BenchmarkXPRT, Brave, Browser-based benchmarks, Chrome OS, Chromium, Cross-platform benchmarks, Firefox, Google Chrome, Microsoft Edge, Mozilla, Opera, Performance benchmarking, results, WebXPRT, WebXPRT 4 | Tagged , , , , , , , , , , , , |

Let us know if you encounter this Adobe PSE 2020 issue with HDXPRT 4

Justin Greene
By Justin Greene
on October 6, 2022

Last week, a member of the tech press let us know that they encountered an error while preparing a system for HDXPRT 4 testing. Specifically, while attempting to install the trial version of Adobe Photoshop Elements (PSE) 2020, they encountered the following error:

Update Required

Your browser or operating system is no longer supported. You may need to install the latest updates to your operating system.

They were working with an MSI Sword 15 A12UE, which had all the latest Windows 11 and Microsoft Edge updates, and they were able to complete installation and testing on other Windows 11 systems in their lab. This eliminates compatibility between the Adobe PSE 2020 installer package and Windows 11 or Microsoft Edge as the issue.

We do not have the same MSI Sword system in our lab, but we tried to replicate the issue by performing the HDXPRT 4 installation and setup process on a Dell G7 15 laptop running on an up-to-date version of Windows 11 (22H2, 22621.521). We successfully installed Adobe PSE 2020 and completed several HDXPRT 4 iterations.

The error this user encountered could be specific to their system or situation. However, we would like to know if other HDXPRT 4 users have run into the same issue. If you’ve experienced this issue in your testing, please contact us. We may be able to identify and publish a solution. 

Justin

Posted in Adobe, Adobe Photoshop, benchmark, BenchmarkXPRT, HDXPRT, HDXPRT 4, Microsoft Edge, Performance benchmarking, Windows 11 | Tagged , , , , , , , |

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2024   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x