BenchmarkXPRT Blog banner

Category: Automation

How to automate your WebXPRT 4 testing

Justin Greene
By Justin Greene
on August 22, 2024

We’re excited about the ongoing upward trend in the number of completed WebXPRT 4 runs that we’re seeing each month. OEM and tech press labs are responsible for a significant amount of that growth, and many of them use WebXPRT’s automation features to complete large blocks of hands-off testing at one time. We realize that many new WebXPRT users may be unfamiliar with the benchmark’s automation tools, so we decided to provide a quick guide to WebXPRT automation in today’s blog. Whether you’re testing one or 1,000 devices, the instructions below can help save you some time.

WebXPRT 4 allows users to run scripts in an automated fashion and control test execution by appending parameters and values to the WebXPRT URL. Three parameters are available:

  • test type
  • test scenarios
  • results

Below, you’ll find a description of those parameters and instructions for how you can use them to automate your test runs.

Test type

The WebXPRT automation framework accounts for two test types: (1) the six core workloads, and (2) any experimental workloads we might add in future builds. There are currently no experimental tests in WebXPRT 4, so always set the test type variable to 1.

  • Core tests: 1

Test scenario

The test scenario parameter lets you specify which subtest workloads to run by using the following codes:

  • Photo enhancement: 1
  • Organize album using AI: 2
  • Stock option pricing: 4
  • Encrypt notes and OCR scan using WASM: 8
  • Sales graphs: 16
  • Online homework: 32

To run a single subtest workload, use its code. To run multiple workloads, use the sum of their codes. For example, to run Stock options pricing (4) and Encrypt notes and OCR scan (8), use the sum of 12. To run all core tests, use 63, the sum of all the individual test codes (1 + 2 + 4 + 8 + 16 + 32 = 63).

Results format

The results format parameter lets you select the format of the results:

  • Display the result as an HTML table: 1
  • Display the result as XML: 2
  • Display the result as CSV: 3
  • Download the result as CSV: 4

To use the automation feature, start with the URL https://www.principledtechnologies.com/benchmarkxprt/webxprt/2021/wx4_build_3_7_3, append a question mark (?), and add the parameters and values separated by ampersands (&). For example, to run all the core tests and download the results, you would use the following URL: https://principledtechnologies.com/benchmarkxprt/webxprt/2021/wx4_build_3_7_3/auto.php?testtype=1&tests=63&result=4

We hope WebXPRT 4’s automation features will make testing easier for you. If you have any questions about WebXPRT or the automation process, please feel free to ask!

Justin

Posted in Automation, benchmark, Benchmark metrics, Benchmarking, browser performance, Browser-based benchmarks, Performance benchmarking, results, tech press, WebXPRT, WebXPRT 4 | Tagged , , , , , , , , , , |

WebXPRT’s mirror host site in Singapore

Justin Greene
By Justin Greene
on May 25, 2023

If you’ve ever spent time exploring WebXPRT.com, you may have noticed a line that says, “If you are in East Asia, you can run WebXPRT from our Singapore host,” followed by a hyperlink with Simplified Chinese characters. We realize that some people may not know why we have a WebXPRT mirror host site in Singapore—or how to use it—so today’s post will cover the basics.

When we first released WebXPRT 2013, some users in mainland China reported slow download times when running the benchmark. These slowdowns affected initial page and workload content load times, but not workload execution, which happens locally. As a result, subtest and overall scores were still consistent with expectations for the devices under test, but it took longer than normal for test runs to complete. In response, we set up a mirror host site in Singapore to facilitate WebXPRT testing in China and other East Asian countries. We continued this practice with subsequent WebXPRT versions, and currently offer Singapore-based instances of WebXPRT 4, WebXPRT 3, and WebXPRT 2015.

The link to WebXPRT 4 Singapore on WebXPRT.com

The default UI language on the Singapore site is Simplified Chinese, but users can opt to change the language to English or German. Apart from a different default language, the WebXPRT mirror instances hosted in Singapore are identical to the instances on the main WebXPRT site. If you test a device on WebXPRT Singapore and WebXPRT.com, you should see similar performance scores from both sites.

The start page for WebXPRT 4 Singapore, with the default Simplified Chinese UI

We hope that the WebXPRT mirror host site in Singapore will make it easier for people in East Asia to use the benchmark. Do you find the site useful? If so, we’d love to hear from you! Also, if you encounter any unexpected issues or interruptions while testing, please let us know!

Justin

Posted in Automation, benchmark, Benchmarking, Browser-based benchmarks, Cross-platform benchmarks, Performance benchmarking, Simplified Chinese, WebXPRT, WebXPRT 2013, WebXPRT 2015, WebXPRT 3, WebXPRT 4 | Tagged , , , , , , , , |

We want your thoughts about experimental WebXPRT 4 workloads

Justin Greene
By Justin Greene
on December 8, 2022

Two weeks ago, we discussed how users can automate WebXPRT 4 testing by appending several parameters and values to the benchmark’s URL. One of these lets you enable any available experimental workloads during the test run. While we don’t currently offer any experimental workloads for WebXPRT 4, we are seeking suggestions for possible future workload scenarios, or specific web technologies that you’d like to be able to test with an experimental workload.

The main purpose of optional, experimental workloads would be to test cutting-edge browser technologies or new use cases, even if the experimental workload doesn’t work on all browsers or devices. The individual scores for the experimental workloads would stand alone, and would not factor in the WebXPRT 4 overall score. WebXPRT 4 testers would be able to run the experimental workloads one of two ways: by adjusting a value in the WebXPRT 4 automation scripts, as mentioned above, or by manually selecting them on the benchmark’s home screen.

Testers would benefit from experimental workloads by learning how well certain browsers or systems handle new tasks (e.g., new web apps or AI capabilities). We would benefit from fielding workloads for large-scale testing and user feedback before we commit to including them as core WebXPRT workloads.

Do you have any general thoughts about experimental workloads for browser performance testing, or any specific workloads that you’d like us to consider? Please let us know.

Justin

Posted in AI, Automation, benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, Future of performance evaluation, Performance benchmarking, WebXPRT, WebXPRT 4, What makes a good benchmark? | Tagged , , , , , , , , |

How to automate WebXPRT 4 testing

Justin Greene
By Justin Greene
on November 23, 2022

As the number of WebXPRT runs continues to grow, we realize many new WebXPRT users may be unfamiliar with all the features and capabilities of the benchmark. To help inform users about features that might facilitate their testing, we’ve decided to highlight a few WebXPRT features here in the blog. A few weeks ago, we discussed the multiple language options available in the WebXPRT 4 UI. This week, we look at WebXPRT 4 test automation.

WebXPRT 4 allows users to run scripts in an automated fashion. You can control the execution of WebXPRT 4 by appending parameters and values to the WebXPRT URL. Three parameters are available: testtype, tests, and result. Below, you’ll find a description of those parameters and instructions for utilizing automation.

Test type

The WebXPRT automation framework accounts for two test types: (1) the six core workloads and (2) any experimental workloads we might add in future builds. There are currently no experimental tests in WebXPRT 4, so always set the test type variable to 1.

  • Core tests: 1

Test scenario

This parameter lets you specify which tests to run by using the following codes:

  • Photo enhancement: 1
  • Organize album using AI: 2
  • Stock option pricing: 4
  • Encrypt notes and OCR scan using WASM: 8
  • Sales graphs: 16
  • Online homework: 32

To run a single individual test, use its code. To run multiple tests, use the sum of their codes. For example, to run Stocks (4) and Notes (8), use the sum of 12. To run all core tests, use 63, the sum of all the individual test codes (1 + 2 + 4 + 8 + 16 + 32 = 63).

Results format

This parameter lets you select the format of the results:

  • Display the result as an HTML table: 1
  • Display the result as XML: 2
  • Display the result as CSV: 3
  • Download the result as CSV: 4

To use the automation feature, start with the URL http://www.principledtechnologies.com/benchmarkxprt/webxprt/2021/wx4_build_3_7_3, append a question mark (?), and add the parameters and values separated by ampersands (&). For example, to run all the core tests and download the results, you would use the following URL: http://principledtechnologies.com/benchmarkxprt/webxprt/2021/wx4_build_3_7_3/auto.php?testtype=1&tests=63&result=4

We hope the WebXPRT automation features will make testing easier for you. If you have any questions about WebXPRT or the automation process, please feel free to ask!

Justin

Posted in Automation, benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, Browser-based benchmarks, Cross-platform benchmarks, Performance benchmarking, results, Web-based testing, WebAssembly, WebXPRT, WebXPRT 4 | Tagged , , , , , , |

The Exploring WebXPRT 4 white paper is now available

Justin Greene
By Justin Greene
on September 1, 2022

This week, we published the Exploring WebXPRT 4 white paper. It describes the design and structure of WebXPRT 4, including detailed information about the benchmark’s harness, HTML5 and WebAssembly (WASM) capability checks, and changes we’ve made to the structure of the performance test workloads. We explain the benchmark’s scoring methodology, how to automate tests, and how to submit results for publication. The white paper also includes information about the third-party functions and libraries that WebXPRT 4 uses during the HTML5 and WASM capability checks and performance workloads.

The Exploring WebXPRT 4 white paper promotes the high level of transparency and disclosure that is a core value of the BenchmarkXPRT Development Community. We’ve always believed that transparency builds trust, and trust is essential for a healthy benchmarking community. That’s why we involve community members in the benchmark development process and disclose how we build our benchmarks and how they work.

You can find the paper on WebXPRT.com and our XPRT white papers page. If you have any questions about WebXPRT 4, please let us know, and be sure to check out our other XPRT white papers.

Justin

Posted in Automation, benchmark, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Performance benchmarking, results, results submission, WebAssembly, WebXPRT, WebXPRT 3, WebXPRT 4, White papers | Tagged , , , , , , , , , |

Thinking about experimental WebXPRT workloads in 2022

Justin Greene
By Justin Greene
on November 18, 2021

As the WebXPRT 4 development process has progressed, we’ve started to discuss the possibility of offering experimental WebXPRT 4 workloads in 2022. These would be optional workloads that test cutting-edge browser technologies or new use cases. The individual scores for the experimental workloads would stand alone, and would not factor in the WebXPRT 4 overall score.

WebXPRT testers would be able to run the experimental workloads one of two ways: by manually selecting them on the benchmark’s home screen, or by adjusting a value in the WebXPRT 4 automation scripts.

Testers would benefit from experimental workloads by being able to compare how well certain browsers or systems handle new tasks (e.g., new web apps or AI capabilities). We would benefit from fielding workloads for large-scale testing and user feedback before we commit to including them as core WebXPRT workloads.

Do you have any general thoughts about experimental workloads for browser performance testing, or any specific workloads that you’d like us to consider? Please let us know.

Justin

Posted in AI, Automation, benchmark, Browser-based benchmarks, Future of performance evaluation, Performance benchmarking, Web-based testing, WebXPRT, webxprt 4, What makes a good benchmark? | Tagged , , , , , , , , |

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2024   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x