Category: Future of performance evaluation

The WebXPRT 4 development process is far enough along that we’d like to share more about changes we are likely to make and a rough target date for publishing a preview build. While some of the details below will probably change, this post should give readers a good sense of what to expect.

General changes

Some of the non-workload changes in WebXPRT 4 relate to our typical benchmark update process, and a few result directly from feedback we received from the WebXPRT tech press survey.

• We will update the aesthetics of the WebXPRT UI to make WebXPRT 4 visually distinct from older versions. We do not anticipate significantly changing the flow of the UI.
• We will update content in some of the workloads to reflect changes in everyday technology. For instance, we will upgrade most of the photos in the photo processing workloads to higher resolutions.
• In response to a request from tech press survey respondents, we are considering adding a looping function to the automation scripts.
• We are investigating the possibility of shortening the benchmark by reducing the default number of iterations from seven to five. We will only make this change if we can ensure that five iterations produce consistently low score variance.

Changes to existing workloads

• Photo Enhancement. This workload applies three effects to two photos each (six photos total). It tests HTML5 Canvas, Canvas 2D, and JavaScript performance. The only change we are considering is adding higher-resolution photos.

• Organize Album Using AI. This workload currently uses the ConvNetJS neural network library to complete two tasks: (1) organizing five images and (2) classifying the five images in an album. We are planning to replace ConvNetJS with WebAssembly (WASM) for both tasks and are considering upgrading the images to higher resolutions.
• Stock Option Pricing. This workload calculates and displays graphic views of a stock portfolio using Canvas, SVG, and dygraph.js. The only change we are considering is combining it with the Sales Graphs workload (below).
• Sales Graphs. This workload provides a web-based application displaying multiple views of sales data. Sales Graphs exercises HTML5 Canvas and SVG performance. The only change we are considering is combining it with the Stock Option Pricing workload (above).
• Encrypt Notes and OCR Scan. This workload uses ASM.js to sync notes, extract text from a scanned receipt using optical character recognition (OCR), and add the scanned text to a spending report. We are planning to replace ASM.js with WASM for the Notes task and with WASM-based Tesseract for the OCR task.
• Online Homework. This workload uses regex, arrays, strings, and Web Workers to review DNA and spell-check an essay. We are not planning to change this workload.

Possible new workloads

• Natural Language Processing (NLP). We are considering the addition of an NLP workload using ONNX Runtime and/or TensorFlowJS. The workload would use Bidirectional Encoder Representations from Transformers (BERT) to answer questions about a given text. Similar use cases are becoming more prevalent in conversational bot systems, domain-specific document search tools, and various other educational applications.
• Message Scrolling. We are considering developing a new workload that would use an Angular or React.js to scroll through hundreds of messages. We’ll share more about this possible workload as we firm up the details.

The release timeline

We hope to publish a WebXPRT 4 preview build in the second half of November, with a general release before the end of the year. If it looks as though that timeline will change significantly, we’ll provide an update here in the blog as soon as possible.

We’re very grateful for all the input we received during the WebXPRT 4 planning process. If you have any questions about the details we’ve shared above, please feel free to ask!

Justin