Category: WebXPRT 2015

Rolling with the changes

on June 4, 2015

While WebXPRT 2015 has been running fine on earlier beta versions of Windows 10, we have found a problem on some recent versions. Starting with build 10122, the Local Notes test hangs when using the Microsoft Edge browser. (Note: This browser still identifies itself as Spartan in the builds we have seen.) Chrome and Firefox on Windows 10 have successfully run WebXPRT 2015, so the problem appears to be restricted to Edge/Spartan.

Because WebXPRT ran successfully on earlier builds of Windows 10, we are hoping that upcoming builds will resolve the problem. However, we have been investigating the issue in case there is something we can address. The problem is that the encrypted strings that the test is trying to write to LocalStorage are not being written correctly. Non-encrypted strings are being written correctly.

As soon as the problem gets resolved, we’ll let you know.

In other news, we’ve been looking at Android M. There are a lot of interesting changes coming, such as the difference in the way that Android M manages app permissions. We’ve decided to delay releasing the design document for the next version of MobileXPRT so that we can make sure that the design deals with these changes appropriately.

Eric

Posted in Android, Benchmarking, Chrome OS, Collaborative benchmark development, MobileXPRT, WebXPRT, WebXPRT 2015, Windows 10 |

It’s not the same

By Eric Hale

on May 21, 2015

We sometimes get questions about comparing results from older versions of benchmarks to the current version. Unfortunately, it’s never safe to compare the results from different versions of benchmarks. This principle has been around much longer than the XPRTs. A major update will use different workloads and test data, and will probably be built with updated or different tools.

To avoid confusion, we rescale the results every time we release a new version of an existing benchmark. By making the results significantly different, we hope to reduce the likelihood that results from two different versions will get mixed together.

As an example, we scaled the results from WebXPRT 2015 to be significantly lower than those from WebXPRT 2013. Here are some scores from the published results for WebXPRT 2013 and WebXPRT 2015.

Please note that the results above are not necessarily from the same device configurations, and are meant only to illustrate the difference in results between the two versions of WebXPRT.

If you have any questions, please let us know.

Eric

Posted in BenchmarkXPRT, Collaborative benchmark development, WebXPRT, WebXPRT 2013 results, WebXPRT 2015, What makes a good benchmark? |

WebXPRT 2015 source code is now available

By Eric Hale

on May 19, 2015

As of today, we are making the WebXPRT 2015 source code available to community members.

Download the WebXPRT 2015 source here (login required).

We’ll also post a link to the source on the WebXPRT tab in the Members Area. The source code package contains instructions for setting up a local installation of WebXPRT. However, please note that you must test using WebXPRT.com to get an official result.

If you want more information, please contact BenchmarkXPRTSupport@principledtechnologies.com.

We look forward to your feedback!

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Source code, WebXPRT 2015 |

WebXPRT 2015 is here!

By Eric Hale

on May 8, 2015

Today, we’re releasing WebXPRT 2015, our benchmark for evaluating the performance of Web-enabled devices. The BenchmarkXPRT Development Community has been using a community preview for several weeks, but now that we’ve released the benchmark, anyone can run WebXPRT and publish results.

Run WebXPRT 2015

WebXPRT 2013 is still available here while people transition to WebXPRT 2015. We will provide plenty of notice before discontinuing WebXPRT 2013.

After trying out WebXPRT, please send your comments to BenchmarkXPRTsupport@principledtechnologies.com.

Posted in Automation, BenchmarkXPRT development community, Performance benchmarking, Performance of computing devices, Performance testing on tablets, WebXPRT, WebXPRT 2015 |

Looking at the data

By Eric Hale

on April 30, 2015

We’re planning the general release of WebXPRT 2015 for late next week. The testing is looking good and the response has been positive.

We’ve been looking at the hundreds of runs in the database for the community preview. As we’ve said before, we’ve been looking at the information from the JavaScript navigator object in the hope that we could improve the disclosure information from WebXPRT 2015. However, the information is not reliable enough for us to use it at this time. Hopefully, that will improve in the near future.

For now, we will continue to use the information from the user agent string. We’ve discussed the user agent string before. It does give us some information about the device, although not as much as we are able to gather in some of the XPRTs.

Looking at the data, the most common OS has been Windows. This may be in part because you needed to be logged in to run the CP. However, Android devices represented over a third of the runs, and Chrome OS represented about 25 percent of the runs. While we had healthy numbers of iOS devices, there were only a handful of Mac OSX runs. Chrome was the most common browser. Other browsers identified themselves as Safari, Firefox, Opera, and MS IE.

As you can see, the new WebXPRT continues the tradition that WebXPRT 2013 started of running everywhere. We can’t wait to make it available to general public!

Eric

Posted in BenchmarkXPRT development community, Community Preview, WebXPRT, WebXPRT 2013 release, WebXPRT 2015 |

Lots of things are happening!

By Eric Hale

on April 9, 2015

The WebXPRT 2015 community preview hasn’t even been out two weeks yet, but there are already hundreds of runs in the database. If you haven’t checked it out yet, now is a good time! (login required)

As I mentioned last week, Bill and Justin are at Intel Developer Forum 2015 – Shenzhen. Here’s their home away from home this week.

Bill and Justin have been having a lot of good conversations and have found a lot of interest in an open development community. We’re looking forward to having even more members in Asia soon!

On Monday we released the CrXPRT white paper. If you want to know more about the concepts behind CrXPRT 2015, how it was developed, how the results are calculated, or anything else about CrXPRT, the white paper is a great place to start.
Finally, the MobileXPRT 2015 design document is coming in the next couple of weeks. What would you like to see in the next version of MobileXPRT? 64-bit support? New types of tests? Improvements to the UI? Everything is on the table. This is the time to make your voice heard!
Eric

Posted in CrXPRT, IDF, Principled Technologies, WebXPRT, WebXPRT 2015, White papers |

Category: WebXPRT 2015

Rolling with the changes

It’s not the same

WebXPRT 2015 source code is now available

WebXPRT 2015 is here!

Looking at the data

Lots of things are happening!

Check out the other XPRTs: