Category: MobileXPRT

Time to Play

on August 15, 2013

As Bill said in MobileXPRT 2013 lives!, we released MobileXPRT 2013 to the public on Monday. Since that time, we have published MobileXPRT as a free app in the Google Play Store. You can find it by searching for “mobilexprt” or you can get it here. Downloading the app from the store simplifies the installation, but in all other regards, it’s the same application that you can download from the MobileXPRT site.

We will make the source for the release available to the community in a few days.

We have some MobileXPRT results on the Web site. Some of these results are from our own testing and some are from published articles like this one from Tom’s Hardware. We will be adding more results from devices we test and looking for more articles as time goes on.

We’d love to add your results as well. You can find out how to submit your results here. If you have a blog or publish your own reviews, send us the link. If we include your results, we will link back to your site.

Now on to the next great thing!

Eric

Comment on this post in the forums

Posted in Android, MobileXPRT |

MobileXPRT 2013 lives!

By Bill Catchings

on August 12, 2013

On behalf of the BenchmarkXPRT Development Community, I’m proud to announce that today we are releasing MobileXPRT 2013. MobileXPRT is our benchmark for measuring the performance of Android-based devices. Like the other benchmarks (or XPRTs as we sometimes call them), MobileXPRT attempts to measure performance by using activities that real users do on their devices. For more information, check out our press release http://www.prweb.com/releases/2013/8/prweb11016543.htm.

We still have more to do over the next few days. We need to release the source, which we expect to do later this week. We need to make MobileXPRT 2013 available on Google Play. We’ll also work to add more results to our MobileXPRT 2013 results database. Watch this space for more details!

I’d also like to thank those of you that have helped us by developing, testing, or commenting on the Community Preview release. Your efforts have been an important part of making this benchmark available to a larger audience. Please continue to let us know your thoughts, send us your results, and tell us about any problems you find.

Benchmark development is never done!

Bill

Comment on this post in the forums

Posted in MobileXPRT |

Fragments

By Eric Hale

on August 1, 2013

The folks over at OpenSignal have created some pretty impressive visualizations of how fragmented the Android marketplace is. Pictures like that are a little daunting as we head to the public release of MobileXPRT 2013. However, the benchmark is looking pretty stable. Since we released MobileXPRT CP1.1 back in June, we have not encountered any problem configurations.

That includes our first encounter with the recently released Android 4.3. We tested MobileXPRT 2013 CP 1.1 on a Nexus 7 running Android 4.3. The benchmark ran with no problem. Here are the scores, along with scores from the same device running Android 4.2.2.

	Android 4.2.2	Android 4.3
MobileXPRT performance	116	120
MobileXPRT user experience	97	98

Of course, WebXPRT 2013 ran on Android 4.3 as well. If you’re curious, here are the scores.

	Android 4.2.2	Android 4.3
WebXPRT	183	185

While the upgrade did not have a big effect on the score in the case of the Nexus 7, we will need to test on more devices before we can make a definitive statement about the effect of Android 4.3.3 on performance.

Eric

Comment on this post in the forums

Posted in MobileXPRT, WebXPRT |

Oh no, not another learning experience!

By Eric Hale

on July 18, 2013

We are in the process of submitting TouchXPRT 2013 to the Microsoft App Store, and we are learning a lot in the process. Of course, we knew to create a Live ID for the process. However, we were surprised by the verification for the corporate account. We had to produce a lot of documentation, including our articles of incorporation, respond to phone calls and e-mail verifications, and agree to a pretty stringent legal agreement. Given the legal weight attached to the Live ID, we keep its information restricted to a minimum number of people.

Because the packages you upload to the store must be digitally signed by the Live account associated with the app store registration, a designated representative has to rebuild the package from the development team before we can submit it.

When we were preparing TouchXPRT 2013 for the app store, we restructured it to download its content after it’s installed. This allows the package in the app store to be smaller. Hosting the content was not a problem, but we did run into a couple of issues.

The first time we submitted TouchXPRT 2013 to the app store, they rejected it because they thought it was incomplete. We added a note to the testers about the download, and that resolved that issue. Next time, of course, we have learned to value of very specific notes to the testers.

The second hurdle was that, now that TouchXPRT downloaded its data, it qualified as a “network aware” application. All network aware applications must have a privacy policy and the privacy policy must be available from the application setting on the charms menu. Simply having the privacy policy was not enough. We have fixed this and submitted the application again.

The privacy policy is very simple: “TouchXPRT 2013 uses the network only to download content for its performance tests. TouchXPRT does not report any information to Principled Technologies or any other entity.” You can find it here.

In the next few weeks, we will be releasing MobileXPRT 2013 to the general public. We are already preparing it to submit to Google Play. Hopefully, things will go smoother this time!

Eric

Comment on this post in the forums

Posted in MobileXPRT, TouchXPRT |

Back to the source

By Eric Hale

on July 4, 2013

Last week, we released MobileXPRT CP1.1.This week, we released the source code for MobileXPRT CP1.1. You can download it here (login required). The procedure for building it is the same as for the previous CP. As we discussed in Kick the tires. Look under the hood, it’s easy to set up the environment and all the necessary software is free.

We believe that one of the most important things we can do is make the source code available. We believe that increasing the transparency of the benchmarking process and stimulating the participation of industry players and the public in the definition, development, understanding, and auditing of the benchmarks will lead to better benchmarks.

You may be thinking “Then why not open source the benchmark?” The short answer is that we need to make sure that the results from any version of our benchmarks are ones you can trust. You can watch Bill discuss this in BenchmarkXPRT: It’s not a benchmark.

We believe that the community model—which gives you total visibility into the benchmark, allows you to run your own experiments and contribute to future versions of the benchmark, and still protects the integrity of the results—strikes the right balance.

If you’re not a member, please consider joining. It’s easy.

If you are a member, check out the code and tell us how it can be better!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT development community, Collaborative benchmark development, MobileXPRT 2013 source, Video |

A new version and a new video

By Eric Hale

on June 27, 2013

As we said in It’s finally here!, the Android ecosystem is very diverse and we knew that MobileXPRT might have problems on some devices. So far, we have encountered a couple of issues:

The Slideshow test failed during post validation check on the Onda tablet.
The Zoom-n-pinch test was unstable, failing part of the time, on the Zopo phone

On Friday, we will be releasing MobileXPRT 2013 community preview 1.1 (CP1.1), which fixes these problems. The results from version CP1.1 are comparable to the current community preview. If you have any problems, or questions, please e-mail us at benchmarkxprtsupport@principledtechnologies.com

We will release the source for CP1.1 next week.

In other news, we released a new video this week, BenchmarkXPRT: It’s not a benchmark. In this video, Bill answers some common questions about the BenchmarkXPRT Development Community. If you’ve ever wondered what exactly BenchmarkXPRT is, or why the world needs more benchmarks, this is the video for you.

Eric

Comment on this post in the forums

Posted in Android, BenchmarkXPRT, BenchmarkXPRT development community, MobileXPRT, MobileXPRT 2013 source, Video |

Category: MobileXPRT

Time to Play

MobileXPRT 2013 lives!

Fragments

Oh no, not another learning experience!

Back to the source

A new version and a new video

Check out the other XPRTs: