BenchmarkXPRT Development Community Blog

Sources

on September 19, 2013

If you’ve checked out the MobileXPRT and WebXPRT pages recently, you’ve probably noticed that the number of results has started to grow. The results are coming from three sources:

Internal testing at PT.
Results submitted by the public.
Results published on the Web. We link back to the source from these results. Results published on-line include results in reports PT publishes for clients and reviews by other parties.

While we are excited about the growing number of results, we do sanity check them. We compare the results with other runs for the same device when available, or with similar devices if not.

The source code for the benchmarks is available, and we encourage experimentation. However, it should go without saying that valid runs must come from the builds of the benchmarks the development community has published. We can’t compare two results generated by different builds.

That being said, if you change the code and get an interesting result, by all means do contact us. You may have discovered something that we’ll want to include in a future version

Keep the results coming and keep experimenting!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, MobileXPRT, TouchXPRT, WebXPRT |

Improvements

By Eric Hale

on September 12, 2013

We recently made some changes to how MobileXPRT installs. Previously, when you installed MobileXPRT from Google Play, it downloaded the user experience (UX) tests from the Principled Technologies servers. This required you to have the Unknown sources option set, which allows installation of non-market apps, even if you installed MobileXPRT from Google Play.

We have removed this restriction. MobileXPRT now installs the UX tests from Google play, so installation is cleaner. However, the UX tests are not intended to be installed outside of MobileXPRT.

An unrelated fix now makes sure that MobileXPRT cleans out all of its content when you uninstall it. This fix required the location of the results file to move. See Submit your MobileXPRT 2013 results! for details.

If you prefer, you can still install MobileXPRT from the PT site. However, when installing from the PT site, you will still need the Unknown sources option set.

We continue looking for ways to improve the benchmarks. If you have any things you’d like to see us improve, please post to the forums or send an e-mail to benchmarkxprtsupport@principledtechnologies.com

Eric

Comment on this post in the forums

Posted in MobileXPRT |

Endurance

By Bill Catchings

on September 5, 2013

In There’s always something new to measure, we proposed several possible tests for the next version of WebXPRT. Of those, battery life testing generated the most interest.

Battery life testing poses a number of challenges. It’s not as simple as making WebXPRT loop. The biggest challenge is that different devices take different measures when the battery runs low. These measures range from dimming the screen, to stopping the hard disk, to totally shutting down the device. While these are perfectly reasonable, they are out of the benchmark’s control. Worse, most current browsers offer no way of knowing that these measures even happen nor do they offer good ways of querying the device to find out the state of its battery. We want to make sure that our approach does not unfairly advantage one device over another and gives a fair and accurate measure.

Because WebXPRT is a hosted application, we are looking at one of the other XPRT benchmarks for our first attempt at adding battery life to an existing benchmark. MobileXPRT seems to be the best fit. It runs on Android, which has a functional API for monitoring and managing power events, and the diversity of the Android ecosystem forces the benchmark to deal with a greater range of devices and OS configurations than TouchXPRT.

We are trying a number of approaches, and we have made some progress. We will discuss what we have learned in the next few weeks.

Our hope is that what we learn from MobileXPRT will better equip us to add battery life testing to WebXPRT.

Have any thoughts or comments? Post to the forums or e-mail benchmarkxprtsupport@principledtechnologies.com to let us know.

-Bill

Comment on this post in the forums

Posted in TouchXPRT, WebXPRT |

Designated Drivers

By Eric Hale

on August 29, 2013

As we mentioned in last week’s blog On to the next thing, we have seen some problems running HDXPRT 2012 on the Windows 8.1 preview, build 9460. To date, the failures we’ve seen have been in Media Espresso’s Power Director on systems using third and fourth generation Intel Core processors.

We are happy to say that HDXPRT 2012 runs fine on the preview of Windows 8.1 when using the Windows 8.1 Preview Beta Graphics Driver. We are continuing to test, but things are looking good. You can get the graphic driver at communities.intel.com/docs/DOC-21232. We tested with version 15.33.10.3214, which is the latest version available as I am writing this.

If you see problems when running HDXPRT 2012, please let us know.

As I said last week, we are pushing forward on the development of HDXPRT 2013. We are looking forward to releasing a preview to the community in the next few weeks.

Eric

Comment on this post in the forums

Posted in HDXPRT, HDXPRT 2012, Windows 8.1 |

On to the next thing

By Eric Hale

on August 22, 2013

Last week, we released MobileXPRT 2013 to the public and published it as a free app on Google Play. On Monday, we will release the source code to the community. It hasn’t been long since we released the source code for MobileXPRT CP 1.1, but it’s an important part of the community model that the source for the current version is available to the community.

While we were putting the finishing touches on MobileXPRT, we’ve been hard at work on HDXPRT 2013. The feedback on HDXPRT made it clear that the benchmark should be smaller, faster, and easier to install. We have been working to keep all the value of the benchmark, and update the workloads to reflect current usage, even as we slim it down.

Speaking of HDXPRT, as we mentioned in The show is in previews, HDXPRT 2012 has issues running on Windows 8.1. However, we have had some success getting HDXPRT to run on Windows 8.1 by using beta drivers from Intel, AMD, and NVIDIA. We are still investigating this, and hope to have a general workaround for this soon.

There’s lots more stuff in the pipeline. Exciting times ahead!

Eric

Comment on this post in the forums

Posted in HDXPRT 2012, HDXPRT 2013, MobileXPRT 2013 source, Windows 8.1 |

Time to Play

By Eric Hale

on August 15, 2013

As Bill said in MobileXPRT 2013 lives!, we released MobileXPRT 2013 to the public on Monday. Since that time, we have published MobileXPRT as a free app in the Google Play Store. You can find it by searching for “mobilexprt” or you can get it here. Downloading the app from the store simplifies the installation, but in all other regards, it’s the same application that you can download from the MobileXPRT site.

We will make the source for the release available to the community in a few days.

We have some MobileXPRT results on the Web site. Some of these results are from our own testing and some are from published articles like this one from Tom’s Hardware. We will be adding more results from devices we test and looking for more articles as time goes on.

We’d love to add your results as well. You can find out how to submit your results here. If you have a blog or publish your own reviews, send us the link. If we include your results, we will link back to your site.

Now on to the next great thing!

Eric

Comment on this post in the forums

Posted in Android, MobileXPRT |

Sources

Improvements

Endurance

Designated Drivers

On to the next thing

Time to Play

Check out the other XPRTs: