Category: Benchmarks in general

Long-lasting benchmarks

on June 29, 2011

While researching the Top500 list for last week’s blog, I ran across an interesting article (http://bits.blogs.nytimes.com/2011/05/09/the-ipad-in-your-hand-as-fast-as-a-supercomputer-of-yore/?ref=technology). Its basic premise is that the iPad 2 has about the same computing power as the Cray 2 supercomputer, the world’s fastest computer in 1985. I’m old enough to remember the Cray 1 and Cray 2 supercomputers with their unique circular shapes. In their day, they were very expensive and, consequently, rare. Only government agencies could afford to buy them. Just getting to see one was a big deal. In stark contrast, I seem to see iPads everywhere.

What was the benchmark for determining this? It was LINPACK, the same benchmark that determined the winner of the Top500 earlier in June. Based on the LINPACK results, I am holding in my hand a device that could rival the most powerful in the world about 25 years ago. Another perspective is that I have a phone faster than the most powerful computer in the world the year I graduated with my CS degree. And, I use it to play Angry Birds… (Picture trying to convince someone in the 80s that one day millions of hand-held Cray 2 supercomputers would be used to catapult exploding birds at annoying oinking pigs.)

One interesting thought from all of this is the power of benchmarks that last over time. While it will be a rare (and rather limited) benchmark that can last as long as LINPACK, it is important for benchmarks to not change too frequently. On the other side of the scale is the desire for a benchmark to keep up with current technology. With HDXPRT, we are aiming for about a year between versions. I’d love to know whether you think that is too long, too short, or about right.

Bill

Comment on this post in the forums

Posted in Benchmarks in general, HDXPRT release cycle, History of benchmarking |

Knowing when to wait

By Bill Catchings

on June 15, 2011

Mark mentioned in his blog entry a few weeks ago that waiting sucks. I think we can all agree with that sentiment. However, an experience I had while in Taipei for Computex made me reevaluate that thinking a bit.

I went jogging one morning in a park near my hotel. It was a relatively small park, just a quarter mile around the pond that took up most of the park. I was one of only a couple people jogging, but the park was full of people. Some were walking around the pond. There also were groups of people doing some form of Tai Chi in various clearings around the pond. The path I was on was narrow. At times, there was no way of getting around the people walking without running into the ones doing Tai Chi. That in turn meant running in place at times. Or, put another way, waiting.

Everyone was polite at the encounters, but the contrast between me jogging and the folks doing Tai Chi was stark. I wanted to run my miles as quickly as possible. Those doing Tai Chi were decidedly not in a rush. They were doing their exercises together with others. The goal was to do them at the proper pace in the proper way.

That got me to thinking about waiting on my computer. (Hey, time to think is one of the main reasons I exercise!) There are times when waiting for a computer infuriates me. Other times, however, the computer is fast enough. Or even too fast, like when I’m trying to scroll down to the right cell in Excel and it jumps down to a whole screen full of empty cells. This phenomenon, of course, relates to benchmarks. Benchmarks should measure those operations that are slow enough to hurt productivity or are downright annoying. There is less value in measuring operations that users don’t have to wait on.

Have you had any thoughts about what makes a good benchmark? Even if you weren’t exercising when you had the thought, please share it with the community.

Bill

Comment on this post in the forums

Posted in Benchmarks in general, HDXPRT metrics, What makes a good benchmark? |

Computex – Taipei

By Bill Catchings

on May 31, 2011

It’s hot and muggy here in Taipei. Just like home in North Carolina!

Weather aside, Taipei is definitely not Raleigh. Taipei is a big city with tall buildings. Right next to the hotel is the Taipei 101 which was the world’s tallest building for a few years. The streets are full of cars and motor scooters. People here walk quickly and purposefully. All of Computex seems to be filled with similar purpose and drive. It reminds me a quite bit of COMDEX in Vegas in its prime. Technology has taken over a city only too glad to embrace that technology. In next week’s blog, I’ll let you know about some of the cool things showing here.

I’ve had some interesting HDXPRT meetings so far. One of them helped me to remember some of the non-technical challenges of a successful benchmark. We’ve mentioned benchmark challenges like reliability (it needs to run when you need it to run) and repeatability (it needs to give similar results—within a few percent—each time you run it). I discussed with folks from one PC performance Web site the importance of a benchmark having some permanence. If the benchmark changes too frequently, you can’t compare the current product with the one you reviewed a couple months ago. With HDXPRT, our goal is an annual cycle. That should allow for comparing to older results while still keeping the benchmark current.

Any folks who may be here in Taipei for Computex, please come on by the Hyatt. We can talk about HDXPRT, benchmarks in general, or what you would most like to see in the future of performance evaluation. If nothing else, come by and escape the humidity! Drop us an email at hdxprt_computex@principledtechnologies.com and set up a time to come on over.

Bill

Comment on this post in the forums

Posted in Benchmarks in general, HDXPRT release cycle |

Putting HDXPRT in some benchmark context

By Mark Van Name

on May 6, 2011

Benchmarks come in many shapes and sizes. Some are extremely small, simple, and focused, while others are large, complex, and cover many aspects of a system. To help position HDXPRT in the world of benchmarks, let me share with you a little taxonomy that Bill and I have long used. No taxonomy is perfect, of course, but we’ve found this one to be very helpful as a general categorization tool.

From the perspective of how benchmarks measure performance, you can divide most of them into three groups.

Inspection tools use highly specialized tests to target very particular parts of a system. Back in the day, lo these many decades ago—okay, it was only two decades, but in dog years two tech decades is like five generations—some groups used a simple no-op loop to measure processor performance. I know, it sounds dumb today, but for a short time many felt it was a legitimate measure of processor clock speed, which is one aspect of performance. Similarly, if you want to know how fast a graphics subsystem could draw a particular kind of line, you could write code to draw lines of that type over and over.

These tools have very limited utility, because they don’t do what real users do, but for people working close to hardware, they can be useful.

Moving closer to the real world, synthetic benchmarks are specially written programs that simulate the kinds of work their developers believe real users are doing. So, if you think your target users are spending all day in email, you could write your own mini email client and time functions in it. These tools definitely move closer to real user work than inspection tools, but they still have the drawback of not actually running the programs real people are using.

Application-based benchmarks take that last step by using real applications, the same programs that users employ in the real world. These benchmarks cause those applications to perform the kinds of actions that real users take, and they time those actions. You can always argue about how representative they are—more on that in a future blog entry, assuming I don’t forget to write it—but they are definitely closer to the real world because they’re using real applications.

With all of that background, HDXPRT becomes easy to classify: it’s an application-based benchmark.

Mark Van Name

Comment on this post in the forums

Posted in Application-based benchmarks, Benchmarks in general, History of benchmarking, Performance benchmarking |

Category: Benchmarks in general

Long-lasting benchmarks

Knowing when to wait

Computex – Taipei

Putting HDXPRT in some benchmark context

Check out the other XPRTs: