BenchmarkXPRT Blog banner

Tag Archives: BenchmarkXPRT Development Community

Gain a deeper understanding of WebXPRT 4 with our results calculation white paper

Justin Greene
By Justin Greene
on October 3, 2024

More people around the world are using WebXPRT 4 now than ever before. It’s exciting to see that growth, which also means that many people are visiting our site and learning about the XPRTs for the first time. Because new visitors may not know how the XPRT family of benchmarks differs from other benchmarking efforts, we occasionally like to revisit the core values of our open development community here in the blog—and show how those values translate into more free resources for you.

One of our primary values is transparency in all our benchmark development and testing processes. We share information about our progress with XPRT users throughout the development process, and we invite people to contribute ideas and feedback along the way. We also publish both the source code of our benchmarks and detailed information about how they work, unlike benchmarks that use a “black box” model.

For WebXPRT 4 users who are interested in knowing more about the nuts and bolts of the benchmark, we offer several information-packed resources, including our focus for today, the WebXPRT 4 results calculation and confidence interval white paper. The white paper explains the WebXPRT 4 confidence interval, how it differs from typical benchmark variability, and the formulas the benchmark uses to calculate the individual workload scenario scores and overall score on the end-of-test results screen. The paper also provides an overview of the statistical methodology that WebXPRT uses to translate raw timings into scores.

In addition to the white paper’s discussion of the results calculation process, we’ve also provided a results calculation spreadsheet that shows the raw data from a sample test run and reproduces the calculations WebXPRT uses to generate both the workload scores and an overall score.

In potential future versions of WebXPRT, it’s likely that we’ll continue to use the same—or very similar—statistical methodologies and results calculation formulas that we’ve documented in the results calculation white paper and spreadsheet. That said, if you have suggestions for how we could improve those methods or formulas—either in part or in whole—please don’t hesitate to contact us. We’re interested in hearing your ideas!

The white paper is available on WebXPRT.com and on our XPRT white papers page. If you have any questions about the paper or spreadsheet, WebXPRT, or the XPRTs in general, please let us know.

Justin

Posted in benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, browser performance, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, Performance benchmarking, results, Source code, WebXPRT, WebXPRT 4, White papers | Also tagged , , , , , , , , , , ,

Putting together a good WebXPRT workload proposal

Justin Greene
By Justin Greene
on July 25, 2024

Recently, we announced that we’re moving forward with the development of a new AI-focused WebXPRT 4 workload. It will be an auxiliary workload, which means that it will run as a separate, optional test, and it won’t affect existing WebXPRT 4 tests or scores. Although the inspiration for this new workload came from internal WebXPRT discussions—and, let’s face it, from the huge increase in importance of AI—we wanted to remind you that we’re always open to hearing your WebXPRT workload ideas. If you’d like to submit proposals for new workloads, you don’t have to follow a formal process. Just contact us, and we’ll start the conversation.

If you do decide to send us a workload proposal, it will be helpful to know the types of parameters that we keep in mind. Below, we discuss some of the key questions we ask when we evaluate new WebXPRT workload ideas.

Will it be relevant and interesting to real users, lab testers, and tech reviewers?

When considering a WebXPRT workload proposal, the first two criteria are simple: is it relevant in real life, and are people interested in the workload? We created WebXPRT to evaluate device performance using web-based tasks that consumers are likely to experience daily, so real-life relevance has always been an essential requirement for us throughout development. There are many technologies, functions, and use cases that we could test in a web environment, but only some are relevant to common applications or usage patterns and are likely to draw the interest of real users, lab testers, and technical reviewers.

Will it have cross-platform support?

Currently, WebXPRT runs on almost any web browser and almost every device that supports a web browser. We would like to keep that level of cross-platform support when we introduce new workloads. However, technical differences in how various browsers execute tasks make it challenging to include certain scenarios without undermining our cross-platform ideal. When considering any workload proposal, one of the first questions we ask is, “Will it work on all the major browsers and operating systems?”

There are special exceptions to this guideline. For instance, we’re still in the early days of browser-based AI, and it’s unlikely that a new browser-based AI workload will run on every major browser. If it’s a particularly compelling idea, such as the AI scenario we’re currently working on, we may consider including it as an auxiliary test.

Will it differentiate performance between different types of devices?

XPRT benchmarks provide users with accurate measures for evaluating how well target systems or technologies perform specific tasks. With a broadly targeted benchmark like WebXPRT, if the workloads are so heavy that most devices can’t handle them or so light that most devices complete them without being taxed, the results will be of little use for helping buyers evaluating systems and making purchasing decisions, OEM labs, and the tech press.

That’s why, with any new WebXPRT workload, we look for a sweet spot with respect to how computationally demanding it will be. We want it to run on a wide range of devices—from low-end devices that are several years old to brand-new high-end devices, and everything in between. We also want users to see a wide range of workload scores and resulting overall scores that accurately reflect the experiences those systems deliver, so they can easily grasp the different performance capabilities of the devices under test.

Will results be consistent and easily replicated?

Finally, WebXPRT workloads should produce scores that consistently fall within an acceptable margin of error and are easily replicated with additional testing or comparable gear. Some web technologies are very sensitive to uncontrollable or unpredictable variables, such as internet speed. A workload that measures one of those technologies would be unlikely to produce results that are consistent and easily replicated.

We hope this post will be useful if you’re thinking about potential new workloads that you’d like to see in WebXPRT. If you have any general thoughts about browser performance testing or specific workload ideas that you’d like us to consider, please let us know.

Justin

Posted in AI, benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, browser performance, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, Performance benchmarking, WebXPRT, WebXPRT 4, What makes a good benchmark? | Also tagged , , , , , , , , , , , ,

Want to know how your device performs? Explore the XPRT results database

Justin Greene
By Justin Greene
on May 2, 2024

If you only recently started using the XPRT benchmarks, you may not know about one of the free resources we offer—the XPRT results database. Our results database currently holds more than 3,650 test results from over 150 sources, including global tech press outlets, OEM labs, and independent testers. It serves as a treasure trove of current and historical performance data across all the XPRT benchmarks and hundreds of devices. You can use these results and the results of the same XPRTs on your device to get a sense of how well your device performs.

We update the results database several times a week, adding selected results from our own internal lab testing, reliable media sources, and end-of-test user submissions. (After you run one of the XPRTs, you can choose to submit the results, but don’t worry—this is opt-in. Your results do not automatically appear in the database.) Before adding a result, we also look at any available system information and evaluate whether the score makes sense and is consistent with general expectations.

There are three primary ways that you can explore the XPRT results database.

The first is by visiting the main BenchmarkXPRT results browser, which displays results entries for all of the XPRT benchmarks in chronological order (see the screenshot below). You can filter the results by selecting a benchmark from the drop-down menu. You can also type values, such as a vendor name (e.g., Dell) or the name of a tech publication (e.g., PCWorld) into the free-form filter field. For results we’ve produced in our lab, clicking “PT” in the Source column takes you to a page with additional configuration information for the test system. For sources outside our lab, clicking the source name takes you to the original article or review that contains the result.

The second way to access our published results is by visiting the results page for an individual XPRT benchmark. Start by going to the page of the benchmark that interests you (e.g., CrXPRT.com) , and looking for the blue View Results button. Clicking the button takes you to a page that displays results for only that benchmark. You can use the free-form filter on the page to filter those results, and you can use the Benchmarks drop-down menu to jump to the other individual XPRT results pages.

The third way to view our results database is with the WebXPRT 4 results viewer. The viewer provides an information-packed, interactive tool with which you can explore data from the curated set of WebXPRT 4 results we’ve published on our site. We’ll discuss the features of the WebXPRT 4 results viewer in more detail in a future post.

You can use any of these approaches to compare the results of an XPRT on your device with our many published results. We hope you’ll take some time to explore the information in our results database and that it proves to be helpful to you. If you have ideas for new features or suggestions for improvement, we’d love to hear from you!

Justin

Posted in benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, Browser-based benchmarks, CrXPRT, History of benchmarking, Performance benchmarking, results, results submission, WebXPRT, WebXPRT 4 | Also tagged , , , , , , , ,

Accessing the WebXPRT 4 source code

Justin Greene
By Justin Greene
on March 7, 2024

If you’re new to the XPRTs, you may not be aware that we provide free access to XPRT benchmark source code. Publishing XPRT source code is part of our commitment to making the XPRT development process as transparent as possible. By allowing interested parties to access and review our source code, we’re encouraging openness and honesty in the benchmarking industry. We’re also inviting constructive feedback that can help ensure that the XPRTs continue to improve and contribute to a level playing field for all the types of products they measure.

While we do offer free access to the XPRT source code, we’ve decided to offer the code upon request instead of using a permanent download link. This approach prevents bots or other malicious actors from downloading the code. It also has the benefit of allowing us to interact with users who are interested in the source code and answer any questions they may have. We’re always keen to learn more about what others are thinking about the XPRTs and the types of work they measure.

We recently received some questions about accessing the WebXPRT 4 source code, which made us realize that we needed to make a clearer way for people to ask for the code. In response, we added a “Request WebXPRT 4 source code” link to the gray Helpful Info box on WebXPRT.com (see it in the screenshot below). Clicking the link will allow you to email the BenchmarkXPRT Support team directly and request the code.

After we receive your request, we’ll send you a secure link to the current WebXPRT 4 build package. For those users who wish to set up a local instance of WebXPRT 4 for their own internal testbeds, the package will contain all the necessary files and installation instructions. We allow folks to set up their own instances for purposes of review, internal testing, or experimentation, but we ask that users publish only test results from the official WebXPRT 4 site.

While we offer free access to XPRT source code, our approach to derivative works differs from some traditional open-source models that encourage developers to change products and even take them in different directions. Because benchmarking requires a product that remains static to enable valid comparisons over time, we allow people to download the source, but we reserve the right to control derivative works. This discourages a situation where someone publishes an unauthorized version of the benchmark and calls it an “XPRT.”

If you have any questions about accessing the WebXPRT 4 source code, let us know!

Justin

Posted in Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Open Source, Performance benchmarking, Source code, WebXPRT, WebXPRT 4 | Also tagged , , , , , , ,

WebXPRT benchmarking tips from the XPRT lab

Justin Greene
By Justin Greene
on February 8, 2024

Occasionally, we receive inquiries from XPRT users asking for help determining why two systems with the same hardware configuration are producing significantly different WebXPRT scores. This can happen for many reasons, including different software stacks, but score variability can also result from different testing behaviors and environments. While some degree of variability is normal, these types of questions provide us with an opportunity to talk about some of the basic benchmarking practices we follow in the XPRT lab to produce the most consistent and reliable scores.

Below, we list a few basic best practices you might find useful in your testing. Most of them relate to evaluating browser performance with WebXPRT, but several of these practices apply to other benchmarks as well.

  • Hardware is not the only important factor: Most people know that different browsers produce different performance scores on the same system. Testers are not, however, always aware of shifts in performance between different versions of the same browser. While most updates don’t have a large impact on performance, a few updates have increased (or even decreased) browser performance by a significant amount. For this reason, it’s always important to record and disclose the extended browser version number for each test run. The same principle applies to any other relevant software.
  • Keep a thorough record of system information: We record detailed information about a test system’s key hardware and software components, including full model and version numbers. This information is not only important for later disclosure if we choose to publish a result, it can also sometimes help to pinpoint system differences that explain why two seemingly identical devices are producing very different scores. We also want people to be able to reproduce our results to the closest extent possible, so that commitment involves recording and disclosing more detail than you’ll find in some tech articles and product reviews.
  • Test with clean images: We typically use an out-of-box (OOB) method for testing new devices in the XPRT lab. OOB testing means that other than running the initial OS and browser version updates that users are likely to run after first turning on the device, we change as little as possible before testing. We want to assess the performance that buyers are likely to see when they first purchase the device and before they install additional software. This is the best way to provide an accurate assessment of the performance retail buyers will experience from their new devices. That said, the OOB method is not appropriate for certain types of testing, such as when you want to compare as close to identical system images as possible, or when you want to remove as much pre-loaded software as possible.
  • Turn off automatic updates: We do our best to eliminate or minimize app and system updates after initial setup. Some vendors are making it more difficult to turn off updates completely, but you should always double-check update settings before testing.
  • Get a baseline for system processes: Depending on the system and the OS, a significant amount of system-level activity can be going on in the background after you turn it on. As much as possible, we like to wait for a stable baseline (idle time) of system activity before kicking off a test. If we start testing immediately after booting the system, we often see higher variance in the first run before the scores start to tighten up.
  • Use more than one data point: Because of natural variance, our standard practice in the XPRT lab is to publish a score that represents the median from three to five runs, if not more. If you run a benchmark only once and the score differs significantly from other published scores, your result could be an outlier that you would not see again under stable testing conditions or over the course of multiple runs.


We hope these tips will help make your testing more accurate. If you have any questions about WebXPRT, the other XPRTs, or benchmarking in general, feel free to ask!

Justin

Posted in benchmark, Benchmark metrics, Benchmarking, Benchmarking computing devices, BenchmarkXPRT, BenchmarkXPRT development community, browser performance, Browser-based benchmarks, Cross-platform benchmarks, Performance benchmarking, Performance of computing devices, WebXPRT, WebXPRT 4 | Also tagged , , , , , ,

A bit of house cleaning at BenchmarkXPRT.com

Justin Greene
By Justin Greene
on August 17, 2023

When we’ve released a new version of an XPRT benchmark app, it’s been our practice for many years to maintain a link to the previous version on the benchmark’s main page. For example, visitors can start on the WebXPRT 4 homepage at WebXPRT.com and follow links to access WebXPRT 3, WebXPRT 2015, and WebXPRT 2013. Historically, we’ve maintained these links because labs and tech reviewers usually take a while to introduce a new benchmark to their testing suite. Continued access to the older benchmarks also allows users to quickly compare new devices to old devices without retesting everything.

That being said, several of the XPRT pages currently contain links to benchmarks that we no longer actively support. Some of those benchmarks still function correctly, and testers occasionally use them, but a few no longer work on the latest versions of the operating systems or browsers that we designed them to test. While we want to continue to provide a way for longtime XPRT users to access legacy XPRTs,  we also want to avoid potential confusion for new users. We believe our best way forward is to archive older tests in a separate part of the site.

In the coming weeks, we’ll be moving several legacy XPRT benchmarks to an archive section of the site. Once the new section is ready, we’ll link to it from the Extras drop-down menu at the top of BenchmarkXPRT.com. The benchmarks will still be available in the archive, but we will not actively support them or directly link to them from the homepages of active XPRTs.

During this process, we’ll move the following benchmarks to the archive section:

  • WebXPRT 2015 and 2013
  • CrXPRT 2015
  • HDXPRT 2014
  • TouchXPRT 2014
  • MobileXPRT 2015 and 2013

If you have any questions or concerns about the archive process or access to legacy XPRTs, please let us know! 

Justin

Posted in benchmark, BenchmarkXPRT, CrXPRT, HDXPRT, HDXPRT 2014, History of benchmarking, MobileXPRT, MobileXPRT 2015, TouchXPRT, TouchXPRT 2014, WebXPRT, WebXPRT 2013, WebXPRT 2015, WebXPRT 4 | Also tagged , , , , , , , , , , ,

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2024   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x