Category: Collaborative benchmark development

HDXPRT’s future

on November 3, 2016

While industry pundits have written many words about the death of the PC, Windows PCs are going through a renaissance. No longer do you just choose between a desktop or a laptop in beige or black. There has been an explosion of choices.

Whether you want a super-thin notebook, a tablet, or a two-in-one device, the market has something to offer. Desktop systems can be small devices on your desk, all-in-ones with the PC built into the monitor, or old-style boxes that sit on the floor. You can go with something inexpensive that will be sufficient for many tasks or invest in a super-powerful PC capable of driving today’s latest VR devices. Or you can get a new Microsoft Surface Studio, an example of the new types of devices entering the PC scene.

The current proliferation of PC choices means that tools that help buyers understand the performance differences between systems are more important than they have been in years. Because HDXPRT is one such tool, we expect demand for it to increase.

We have many tasks ahead of us as we prepare for this increased demand. The first is to release a version of HDXPRT 2014 that doesn’t require a patch. We are working on that and should have something ready later this month.

For the other tasks, we need your input. We believe we need to update HDXPRT to reflect the world of high-definition content. It’s tempting to simply change the name to UHDXPRT, but this was our first XPRT and I’m partial to the original name. How about you?

As far as tests, what should a 2017 version of HDXPRT include? We think 4K-related workloads are a must, but aren’t sure whether 4K playback tests are the way to go. What do you think? We need to update other content, such as photo and video resolutions, and replace outdated applications with current versions. Would a VR test would be worthwhile?

Please share your thoughts with us over the coming weeks as we put together a plan for the next version of HDXPRT!

Bill

Posted in Application-based benchmarks, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, HDXPRT, HDXPRT development process, Performance benchmarking, What makes a good benchmark?, Windows 10 |

Taking a detour

By Eric Hale

on October 13, 2016

Back in April, Bill announced that we would be starting development of a cross-platform benchmark. This announcement generated a lot of interest and we got lots of good feedback and ideas.

We knew from the start that getting a cross-platform benchmark right would be hard. However, it proved to be even trickier than we thought. As I explained before, benchmarks not only have to run well, but the results must be fair to all platforms involved. Achieving both of these requirements has been a challenge.

At the same time we’ve been devoting a great deal of effort and resources to the cross-platform benchmark, some increasingly popular new use cases have been receiving less attention than they deserve. We’ve decided that the cross-platform benchmark is not the best use of the Community’s resources, and are going to put it on the shelf for a while. This will free up the resources to let us really dig into some newer technologies.

Thanks again to everyone who responded.

Eric

Posted in Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Cross-platform benchmarks |

Rebalancing our portfolio

By Bill Catchings

on September 29, 2016

We’ve written recently about the many new ways people are using their devices, the growing breadth of types of devices, and how application environments also are changing. We’ve been thinking a lot about the ways benchmarks need to adapt and what new tests we should be developing.

As part of this process, we’re reviewing the XPRT portfolio. An example we wrote about recently was Google’s statement that they are bringing Android apps to Chrome OS and moving away from Chrome apps. Assuming the plan comes to fruition, it has big implications for CrXPRT, and possibly for WebXPRT as well. Another example is that once upon a time, HDXPRT included video playback tests. The increasing importance of 4K video might mean we should bring them back.

As always, we’re interested in your thoughts. Which tests do you see as the most useful going forward? Which ones do you think might be past their prime? What new areas do you like to see us start to address? Let us know!

Over the coming weeks, we’ll share our conclusions based on these market forces and your feedback. We’re excited about the possibilities and hope you are as well.

Bill

Posted in Android, Application-based benchmarks, BenchmarkXPRT development community, Chrome OS, Collaborative benchmark development, CrXPRT, Future of performance evaluation, HDXPRT, WebXPRT, What makes a good benchmark? |

An anniversary update

By Justin Greene

on July 21, 2016

The Windows 10 Anniversary Update release is scheduled for August 2, and we’ve been running the XPRTs on the Windows Insider preview builds. While we can’t publish performance data from developer builds, we’re happy to say that WebXPRT and TouchXPRT run well on the Anniversary Update.

The story for HDXPRT 2014 is more complicated. Back in May, we reported that it would not run on more recent versions of Windows. However, we’ve identified steps that enable HDXPRT to run on the current stable Windows 10 build, as well as the latest Anniversary Update preview. It’s running well, but it’s possible that testers will encounter other issues as Microsoft releases new builds.

We have included the steps below. We’re considering an update to HDXPRT 2014 that will incorporate these changes. If you have any comments or suggestions related to HDXPRT, please let us know.

Justin

Summary
In addition to the normal system configuration requirements for HDXPRT, testers must also overwrite HDXPRT’s CPU-Z files with newer versions and change the default browser from Microsoft Edge to Internet Explorer. After configuring the system for HDXPRT testing, testers may encounter errors related to administrative privileges when attempting to launch Microsoft Edge. Returning User Account Control settings to their default pre-configuration state resolves the problem.

Process
1. Install the latest version of CPU-Z.
      a. Open any browser and download the latest version of CPU-Z for Windows
          (currently CPU-Z 1.76).
      b. Install CPU-Z on the system, using the default settings and installation path.
2. Install the HDXPRT 2014 benchmark using the default installation process. Reboot the system
    after installation.
3. Copy all the files from the C:\Program Files\CPUID\CPU-Z\ directory to the C:\Program Files
    (x86)\HDXPRT\bin, and overwrite the existing CPU-Z files.
4. Change the default browser from Microsoft Edge to Internet Explorer:
      a. Open the Windows Settings app and select System/Default apps.
      b. Under Web browser, click the Edge icon, and select Internet Explorer from the list.
      c. At the Before you switch window, click Switch anyway.
      d. Close the Settings app.
5. Adjust SmartScreen and security settings:
      a. Open Internet Explorer.
      b. Go to Settings/Internet options/Security, and make the following changes for the Internet
           and Trusted Sites zones:
            i. Select Custom Level.
            ii. Disable SmartScreen Filter.
            iii. Under Launching applications and unsafe files, click Enable (not Secure).
            iv. Click OK, and click Apply. If a warning message appears, click Yes.
6. Restart the system.
7. Open HDXPRT and run the benchmark normally.

If, after installing HDXPRT, you encounter an error related to administrative permissions when trying to open Microsoft Edge, return User Account Controls to the default setting, and restart the system. The default User Account Control setting is the third notch from the bottom: “Notify me only when apps try to makes changes to my computer.”

Posted in BenchmarkXPRT development community, Collaborative benchmark development, HDXPRT, HDXPRT 2014, Performance benchmarking, TouchXPRT, Uncategorized, WebXPRT 2015, Windows 10, Windows 10 Anniversary Update |

Open source?

By Bill Catchings

on July 14, 2016

We’re proud of the BenchmarkXPRT Development Community and its accomplishments over the last five years. We’re also thankful for the contributions the members of the community have made. One of the benefits of membership is access to the source code for all the XPRT performance tools. This has meant that the code is available to anyone willing to take the easy step of joining the community.

Behind our decision to use this model rather than a more traditional, open-source model was the need to control derivative works. The license agreement for the source allows members to modify the source, but not to claim that the results from that derivative code are XPRT results. For example, as a member, you may download the TouchXPRT source and modify the workloads for your specific purposes, but you can’t refer to the results as TouchXPRT results.

After much thought and discussion, we have come to believe that we can protect the benchmarks’ reputation within a traditional, open-source framework. While our original concerns are still valid, we think that the success and stature of the XPRTs is such that we can make it available via open source.

However, before we take this step, we want to hear the thoughts, concerns, and opinions of both our community members and the wider public.

Please note that if we do make the code open source, the other benefits of being a member—access to requests for comment, design documents, and community previews—will not change.

Please let us know that you think. Email us or contact us on Twitter.

Bill

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Open Source, TouchXPRT, What makes a good benchmark? |

Seeing the future

By Eric Hale

on July 7, 2016

Back in April we wrote about how Bill’s trip to IDF16 in Shenzhen got us thinking about future benchmarks. Technologies like virtual reality, the Internet of things, and computer vision are going to open up lots of new applications.

Yesterday I saw an amazing article that talked about an automatic computer vision system that is able to detect early-stage esophageal cancer from endoscopy images. These lesions can be difficult for physicians to detect, and the system did very well when compared to four experts who participated in the test. The article contains a link to the original study, for those of you who want more detail.

To me, this is the stuff of science fiction. It’s a very impressive accomplishment. Clearly, new technologies are going to lead to many new and exciting applications.

While this type of application is more specialized than the typical XPRT, things like this get us really excited about the possibilities for the future. Have you seen an application that impressed you recently? Let us know!

Eric

Posted in Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, computer vision, Future of performance evaluation, IDF, Internet of things, IoT, Virtual reality, What makes a good benchmark? |

Category: Collaborative benchmark development

HDXPRT’s future

Taking a detour

Rebalancing our portfolio

An anniversary update

Open source?

Seeing the future

Check out the other XPRTs: