Category: Collaborative benchmark development

The TouchXPRT 2016 CP arrives tomorrow!

on November 12, 2015

As we said a couple of weeks ago, we wanted to test on Windows 10 Threshold 2 before releasing the TouchXPRT 2016 community preview. Well, Threshold 2 is out and the testing has been going very well.

We’ll release the TouchXPRT 2016 to the community tomorrow. Because community previews are not available to the general public, members will need to download it from our site.

The installation procedure is fairly straightforward. First, you put the device in developer mode. Then, for a tablet or PC, run a PowerShell script, as you did for TouchXPRT 2014. For a mobile device, once it’s in developer mode, copy the bundle to the device and install it.

If you have any problems, please let us know.

We’re looking forward to seeing the results from phones and, as we have done with MobileXPRT, to comparing results across different-sized devices.

Enjoy!

Eric

Posted in Benchmark metrics, Benchmarking, Collaborative benchmark development, Community Preview, TouchXPRT, TouchXPRT 2016, TouchXPRT development process, Windows 10 |

Things are looking good!

By Eric Hale

on October 29, 2015

It’s been a couple of weeks since we talked about TouchXPRT 2016. I’m happy to say that we have made great progress since then.

The UI is adapting to the different form factors very well. We’ve tested the resizing on phones and on tablets with the screen snapped to half and quarter screen. While a couple of UI elements still need work, we’ve had no problems running the tests. Here’s how TouchXPRT looks on a phone:

We have also greatly simplified installing the application on phones, so we’re no longer concerned about that.

All in all, things are looking good for releasing the community preview soon. However, we’re going to wait until we can test on the Windows 10 Fall Update (Threshold Build 2). That’s supposed to be released on November 2. We’re not expecting any problems with the Fall Update, but it’s always to wise to check.

November can’t come quickly enough!

Eric

Posted in Benchmarking, BenchmarkXPRT, Collaborative benchmark development, Community Preview, TouchXPRT, TouchXPRT 2016, TouchXPRT development process, TouchXPRT release cycle |

Question we get a lot

By Eric Hale

on October 2, 2015

“How come your benchmark ranks devices differently than [insert other benchmark here]?” It’s a fair question, and the reason is that each benchmark has its own emphasis and tests different things. When you think about it, it would be unusual if all benchmarks did agree.

To illustrate the phenomenon, consider this excerpt from a recent browser shootout in VentureBeat:

While this looks very confusing, the simple explanation is that the different benchmarks are testing different things. To begin with, SunSpider, Octane, JetStream, PeaceKeeper, and Kraken all measure JavaScript performance. Oort Online measures WebGL performance. WebXPRT measures both JavaScript and HTML 5 performance. HTML5Test measures HTML5 compliance.

Even with benchmarks that test the same aspect of browser performance, the tests differ. Kraken and SunSpider both test the speed of JavaScript math, string, and graphics operations in isolation, but run different sets of tests to do so. PeaceKeeper profiles the JavaScript from sites such as YouTube and FaceBook.

WebXPRT, like the other XPRTs, uses scenarios that model the types of work people do with their devices.

It’s no surprise that the order changes depending on which aspect of the Web experience you emphasize, in much the same way that the most fuel-efficient cars might not be the ones with the best acceleration.

This is a bigger topic than we can deal with in a single blog post, and we’ll examine it more in the future.

Eric

Posted in Benchmark metrics, Benchmarking, Benchmarking computing devices, BenchmarkXPRT development community, Collaborative benchmark development, Performance benchmarking, Performance of computing devices, WebXPRT, What makes a good benchmark? |

The benefits of membership

By Eric Hale

on September 24, 2015

We have a couple of goodies for community members coming tomorrow.

The TouchXPRT 2016 design overview will tell you what we’re planning for the upcoming community preview. Thanks to everyone who’s contributed ideas. Let us know if the design overview omits anything you’d like to see in the benchmark.

The MobileXPRT 2015 source code and the instructions for building MobileXPRT 2015 will be available as well. Community members have access to the source for all the XPRT benchmarks. Making the source available is a pillar of the community model.

Look for the design overview and source code in the members’ area.

If you aren’t yet a member, this is a great time to join!

Eric

Posted in Collaborative benchmark development, MobileXPRT, MobileXPRT 2015, TouchXPRT, TouchXPRT 2016, TouchXPRT development process, TouchXPRT release cycle |

Impressed and Excited

By Eric Hale

on September 17, 2015

A couple of weeks ago we talked about some initiatives we’ve been exploring. This week, we’re happy to be able to talk about a new project. We’re sponsoring a senior project with the computer science department at North Carolina State University (NCSU).

As part of their education, small teams of seniors work with local companies on various programming projects. The students are expected to put significant time into these projects, and the tasks aren’t easy. You can get a sense of the range and complexity of these projects by looking at past project proposals.

We submitted a proposal for creating an experimental benchmark test to NCSU and they accepted it and assigned us a team of students. I’ve met a couple of times now with them. I’m impressed and very excited about what they’re going to do. We’re not ready to talk about the project yet, but if they’re successful, we’ll make the new test available on the BenchmarkXPRT site. We might even include it in a future XPRT!

Our hope is that if this project is successful, we can replicate it at other schools and help train the next generation of benchmark developers. As a bonus, the BenchmarkXPRT community will get some fresh perspectives and some new experimental test tools.

Eric

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation |

BenchmarkXPRT in China

By Eric Hale

on September 10, 2015

Last week, we talked about some of the changes we’re making to the BenchmarkXPRT site to make it easier to use. This week, we’d like to talk a bit about improvements we’ve been making to support our users in China.

As you may remember, the first of the XPRTs to have a Chinese UI was BatteryXPRT. We’ve since released WebXPRT 2015 and MobileXPRT 2015, both of which have also have Chinese UIs. We’re also in the process of getting MobileXPRT 2015 listed in several major Chinese app stores. (MobileXPRT 2013 is currently available from Xiaomi and Zhushou 360.)

In other words, we’re always thinking of ways to enhance the XPRT experience for our users in China. To improve download speeds, we’ve long hosted WebXPRT on a mirror site in Singapore. Recently, based on feedback from our users and our own analysis, we’ve changed the way that the privacy notice is displayed on that site. The change allows you to run WebXPRT without loading any Google analytics, which means faster load times for all users.

We will continue to work to improve our localization. This is an area where we can use the help of the community. If you have translation skills and want to contribute the strings for a UI in your language, let us know.

Eric

Last week in the XPRTs

We added a new TouchXPRT result
We added a new HDXPRT result

Posted in BatteryXPRT 2014 for Android, BenchmarkXPRT, BenchmarkXPRT development community, Chinese, Collaborative benchmark development, MobileXPRT, MobileXPRT 2013 source, MobileXPRT 2015, WebXPRT 2015 |

Category: Collaborative benchmark development

The TouchXPRT 2016 CP arrives tomorrow!

Things are looking good!

Question we get a lot

The benefits of membership

Impressed and Excited

BenchmarkXPRT in China

Last week in the XPRTs

Check out the other XPRTs: