BenchmarkXPRT Blog banner

Category: Cross-platform benchmarks

Glimpses of the next WebXPRT

Development work on the next version of WebXPRT is well underway, and we think it’s a good time to offer a glimpse of what’s to come.

We’ve updated the photo-related workloads with new images and are experimenting with adding a new task to the Organize Album workload. The task utilizes ConvNetJS, a JavaScript library designed for training neural networks within the browser itself, to assign classifications to a set of images. It’s an example of the type of integrated deep learning tasks that will be showing up in all sorts of devices in the years to come.

We’re also testing an additional task in the Local Notes workload using Tesseract.js, a popular OCR (optical character recognition) engine. Our scenario uses OCR technology to scan images of purchase receipts and gather relevant information.

We’re testing these new tasks now, and will include them only once we’re confident that they produce consistent and reliable results without extending the benchmark’s runtime unnecessarily.  Consequently, the next WebXPRT might contain variations of these tasks, or other new technologies altogether. We mention them now to offer some insight into the types of workload enhancements that we’re considering.

We’ve been working hard on the new WebXPRT UI as well. The image below shows the new start page from an early development build. We’re still making adjustments, so the final product will probably differ, but you do get a sense of the new UI’s clean look.

WebXPRT screen shot

As we’ve said before, we’re committed to making sure that WebXPRT runs in most browsers and produces results that are useful for comparing web browsing performance across a wide variety of devices. We appreciate the feedback we’ve gotten so far, and are happy to receive more. Do you have ideas for the next WebXPRT? Let us know!

Justin

Introducing the WebXPRT 2015 Processor Comparison Chart

Today, we’re excited to announce the WebXPRT 2015 Processor Comparison Chart, a new tool that makes it easier to access hundreds of PT-curated, real-world performance scores from a wide range of devices covering everything from TVs to phones to tablets to PCs.

The chart offers a quick way to browse and compare WebXPRT 2015 results grouped by processor. Unlike benchmark-score charts that may contain results from unknown sources, PT hand-selected each of the results from internal lab testing and reliable tech media sources. If we published multiple scores for an individual processor, the score presented in the chart will be an average of those scores. Users can hover over and click individual score bars for additional information about the test results and test sources for each processor.

WebXPRT proc chart capture

We think the WebXPRT Processor Comparison Chart will be a valuable resource for folks interested in performance testing and product evaluation, but the current iteration is only the beginning. We plan to add additional capabilities on a regular basis, such a detailed filtering and enhanced viewing and navigational options. It’s also possible that we may integrate other XPRT benchmarks down the road.

Most importantly, we want the chart to be a great asset for its users, and that’s where you come in. We’d love to hear your feedback on the features and types of data you’d like to see. If you have suggestions, please let us know!

Justin

Taking a detour

Back in April, Bill announced that we would be starting development of a cross-platform benchmark. This announcement generated a lot of interest and we got lots of good feedback and ideas.

We knew from the start that getting a cross-platform benchmark right would be hard. However, it proved to be even trickier than we thought. As I explained before, benchmarks not only have to run well, but the results must be fair to all platforms involved. Achieving both of these requirements has been a challenge.

At the same time we’ve been devoting a great deal of effort and resources to the cross-platform benchmark, some increasingly popular new use cases have been receiving less attention than they deserve. We’ve decided that the cross-platform benchmark is not the best use of the Community’s resources, and are going to put it on the shelf for a while. This will free up the resources to let us really dig into some newer technologies.

Thanks again to everyone who responded.

Eric

Side by side

We’re making a big upgrade to the Spotlight: It will now let you make side-by-side comparisons of any of the devices we’ve featured. You’ll also be able to filter devices by type.

We’ve also added pricing information to the Spotlight. The price of a device can vary a lot, depending on the options you select. We give you the price we paid for the device as we tested it. Because prices can vary over time, we also give the date when we bought the device.

We think these changes will make the Spotlight an even more powerful tool for helping with buying decisions.

We also created a special page, the Back to School Roundup. It provides side-by-side comparisons of 13 of the hottest devices to help families and students choose the right ones for their educational needs. We include a range of device types—all-in-ones, Chromebooks, convertibles, laptops, and tablets—and operating systems—Chrome OS, OS X, and Windows.

Going forward, we’ll be adding more advanced querying features. Of course, we’ll also add new devices every week.

Check it out! And, if you have any ideas about how to make the XPRT Weekly Tech Spotlight more useful, let us know!

Eric

Feedback

We’re excited by the high level of interest the community and vendors have shown in the upcoming cross-platform MobileXPRT benchmark. We’ve received general observations about what a cross-platform benchmark should be, along with detailed suggestions about tests, subsystems, and benchmark architecture. We appreciate all of the responses and welcome more, so please keep them coming!

The number-one concern we’ve heard is that we be sure the benchmark tests all platforms fairly. Transparency will be essential to assure users that the tests are performing the same work on all platforms and performing the work in the appropriate way for each platform.

Fortunately, the XPRTs are well positioned to address that concern. From the beginning, we have used a community model. The source code is available to all members, which is the ultimate in transparency.  (If you’re not a community member, it’s easy to join!)

Speaking of source code, we released TouchXPRT source code to the community this week. Members can download the source here (login required).

Eric

Looking ahead

It’s only been a couple of weeks since we announced a cross-platform XPRT. It’s still early days, but we’ve already started getting ideas from vendors and media—from both people within the community and those who have not yet joined. We’re incorporating these ideas into our investigations, and plan to be sending a design document for the community to critique in a few weeks.

However, we are always looking ahead and Bill’s trip to IDF16 got us thinking about future benchmarks. Virtual reality is obviously going to be big. Bill said that he thinks he saw more things using the Oculus Rift than there are Oculus Rifts in the world! The Internet of Things has been ramping up for a while now, and shows no sign of slowing down. Computer vision is another emerging area, one with many possible applications. There are a lot of exciting possibilities!

As always, we want to know what you think. What upcoming technologies are you excited about? What would like to see in these benchmarks? Please let us know!

Eric

Check out the other XPRTs:

Forgot your password?