Category: What makes a good benchmark?

Seeing the future

on July 7, 2016

Back in April we wrote about how Bill’s trip to IDF16 in Shenzhen got us thinking about future benchmarks. Technologies like virtual reality, the Internet of things, and computer vision are going to open up lots of new applications.

Yesterday I saw an amazing article that talked about an automatic computer vision system that is able to detect early-stage esophageal cancer from endoscopy images. These lesions can be difficult for physicians to detect, and the system did very well when compared to four experts who participated in the test. The article contains a link to the original study, for those of you who want more detail.

To me, this is the stuff of science fiction. It’s a very impressive accomplishment. Clearly, new technologies are going to lead to many new and exciting applications.

While this type of application is more specialized than the typical XPRT, things like this get us really excited about the possibilities for the future. Have you seen an application that impressed you recently? Let us know!

Eric

Posted in Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, computer vision, Future of performance evaluation, IDF, Internet of things, IoT, Virtual reality, What makes a good benchmark? |

Watching students become masters

By Eric Hale

on June 23, 2016

As you know, last year, PT sponsored a senior project at the Senior Design Center of North Carolina State University (NCSU). The students created Nebula Wolf, a mini game that might evolve into a future benchmark test. It was a valuable collaboration for us and a very educational experience for the students involved.

I’ve talked before about the emerging technologies we’re considering for new benchmarks. Today, I met with the folks at the NCSU Senior Design Center to discuss a possible future project. We’re hoping to harness the immense energy of these students by having them explore one of these new technologies, and then build on what they discover. Nothing is set yet, but we will, as always, keep you informed as things develop.

We’ll be sharing some exciting news about the XPRT Women Code-a-Thon tomorrow. Check back to find out more! Meanwhile, we hope you enjoy as much as we did the University of Washington Tacoma article on student Viveret, the first place winner of the XPRT Women Code-a-Thon.

Eric

Posted in Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, Nebula Wolf, North Carolina State University, What makes a good benchmark?, XPRT Women Code-a-Thon |

Personal preference

By Eric Hale

on May 19, 2016

I saw an interesting article recently, Here’s why I gave up my beloved Galaxy S7 for a boring old iPhone. It’s only been a few weeks since we featured the Samsung S7 in the XPRT Weekly Tech Spotlight, so of course I had to read it. The interesting thing is this guy really loved his Samsung S7, and even declared it “the best smartphone I’ve ever used.” He loved its VR capabilities, camera, and its look. He even prefers Android as an operating system.

So why would he give it up for an iPhone 6s Plus? Simply put, battery life. As a self-described heavy user, he found his Samsung S7 dying before 5 PM every day. The iPhone 6s Plus lasted much longer.

This is a good reminder that people have different priorities. Your priority could be having the fastest phone, the longest battery life, the best screen, or the broadest compatibility. This is why there is no such this as “the best device.”

This is why we are always asking for your input. Knowing your priorities helps the community build better tests!

Eric

Posted in Android, Battery life, BenchmarkXPRT development community, Collaborative benchmark development, Phones, What makes a good benchmark?, XPRT Weekly Tech Spotlight |

Feedback

By Eric Hale

on April 28, 2016

We’re excited by the high level of interest the community and vendors have shown in the upcoming cross-platform MobileXPRT benchmark. We’ve received general observations about what a cross-platform benchmark should be, along with detailed suggestions about tests, subsystems, and benchmark architecture. We appreciate all of the responses and welcome more, so please keep them coming!

The number-one concern we’ve heard is that we be sure the benchmark tests all platforms fairly. Transparency will be essential to assure users that the tests are performing the same work on all platforms and performing the work in the appropriate way for each platform.

Fortunately, the XPRTs are well positioned to address that concern. From the beginning, we have used a community model. The source code is available to all members, which is the ultimate in transparency. (If you’re not a community member, it’s easy to join!)

Speaking of source code, we released TouchXPRT source code to the community this week. Members can download the source here (login required).

Eric

Posted in Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Cross-platform benchmarks, Future of performance evaluation, MobileXPRT, Performance benchmarking, What makes a good benchmark? |

Women develop new perspectives for the XPRTs

By Jennie Faries

on March 18, 2016

Last weekend, we had the great privilege of co-hosting the first XPRT Women Code-a-Thon with the Seattle chapter of ChickTech. We couldn’t be happier with the results!

Our goal was to bring together a group of women and invite them to develop ideas for new device workloads—workloads that we might include in future versions of MobileXPRT and WebXPRT. The 20 participants—some working individually, and others working as teams—not only met that goal, they did a great deal more.

On the coding front, the participants achieved an impressive amount of work in a very short time. Though we awarded only three prizes, everyone generated interesting and useful ideas. Our prizes went to the following people:

1st place: Viveret Steele, for a 3D-modeling workload

2nd place: Annmarie Aidoo, for a geolocation workload

3rd place: Molly Fallen and Alex Trimble, for an audio-enhancement workload

These four people went home with checks, but winning wasn’t what motivated anyone to participate. Everyone was excited about developing software and working with others. The social side of the event proved to be as meaningful as the technical. People talked, formed friendships and mentoring relationships, and discussed seeking other events like this one. Two people said the event changed their lives.

In the weeks ahead, we’ll be sharing some more information about the event. In the meantime, we’re proud to have been part of it.

Jennie Faries

Last week in the XPRTs

We published the XPRT Weekly Tech Spotlight on the Microsoft Surface 3.
We added two new CrXPRT ’15 results.
We added two new MobileXPRT ’13 results.
We added six new WebXPRT ’15 results.

Posted in BenchmarkXPRT, BenchmarkXPRT development community, ChickTech Seattle, Collaborative benchmark development, Future of performance evaluation, MobileXPRT, WebXPRT, What makes a good benchmark?, XPRT Women Code-a-Thon |

Is it hot in here?

By Eric Hale

on March 3, 2016

One of the great meetings I had at CES was with another community member, Patrick Chang, Senior System Engineer in the Tablet Performance group at Dell. I was glad to hear that, when he tests his devices, Patrick makes frequent use of TouchXPRT.

While TouchXPRT stresses the system appropriately, Patrick’s job requires him to understand not only how well the device performs, but why it performs that way. He was wondering what we could do to help him correlate the temperature, power consumption, and performance of a device.

That information is not typically available to software and apps like the XPRTs. However, it may be possible to add some hooks that would let the XPRTs coordinate with third-party utilities and hardware that do.

As always, the input from the community guides the design of the XPRTs. So, we’d love to know how much interest the community has in having this type of information. If you have thoughts about this, or other kinds of information you’d like the XPRTs to gather, please let us know!

Eric

Last week in the XPRTs
We published the XPRT Weekly Tech Spotlight on the Apple iPad Pro.
We added one new BatteryXPRT ’14 result.
We added one new CrXPRT ’15 result.
We added one new MobileXPRT ’13 result.
We added four new WebXPRT ’15 results.

Posted in Benchmark metrics, Benchmarking, BenchmarkXPRT, CES, Collaborative benchmark development, Dell, Future of performance evaluation, Patrick Chang, Performance benchmarking, Power consumption, Temperature, TouchXPRT, Trade Shows, What makes a good benchmark? |

Category: What makes a good benchmark?

Seeing the future

Watching students become masters

Personal preference

Feedback

Women develop new perspectives for the XPRTs

Is it hot in here?

Check out the other XPRTs: