Category: Benchmarking

A necessary update for HDXPRT 4

on October 17, 2019

If you tried to install HDXPRT 4 over the past few days, you likely noticed that Adobe Photoshop Elements 2018, the version the Edit Photos scenario uses, is no longer available on the Adobe Photoshop Elements download page. In the past, Adobe has provided access to multiple older versions of their software for some time after a new release, but they appear to be moving away from that practice. We have not yet found an alternative way for users to download PSE 2018 on a trial basis. Unfortunately, this means testers will be temporarily unable to successfully complete the HDXPRT 4 installation process.

We’re adapting the scripts in the HDXPRT 4 Edit Photos scenario to use PSE 2020. As soon as we finish, we’ll start testing, with a focus on determining whether the change significantly affects the individual workload or overall scores.

We apologize for the inconvenience that this issue causes for HDXPRT testers. We’ll continue to update the community here in the blog about our progress with the new build. If you have any questions or comments, please let us know.

Justin

Posted in Application-based benchmarks, Benchmarking, BenchmarkXPRT, HDXPRT, HDXPRT 4, HDXPRT capabilities, HDXPRT workloads, Windows | Tagged Adobe, benchmark, HDXPRT, photo editing, Photoshop, Photoshop Elements, Windows |

How to use alternate configuration files with AIXPRT

By Justin Greene

on October 10, 2019

In last week’s AIXPRT Community Preview 3 announcement, we mentioned the new public GitHub repository that we’re using to publish AIXPRT-related information and resources. In addition to the installation readmes for each AIXPRT installation package, the repository contains a selection of alternative test config files that testers can use to quickly and easily change a test’s parameters.

As we discussed in previous blog entries about batch size, levels of precision, and number of concurrent instances, AIXPRT testers can adjust each of these key variables by editing the JSON file in the AIXPRT/Config directory. While the process is straightforward, editing each of the variables in a config file can take some time, and testers don’t always know the appropriate values for their system. To address both of these issues, we are offering a selection of alternative config files that testers can download and drop into the AIXPRT/Config directory.

In the GitHub repository, we’ve organized the available config files first by operating system (Linux_Ubuntu and Windows) and then by vendor (All, Intel, and NVIDIA). Within each section, testers will find preconfigured JSON files set up for several scenarios, such as running with multiple concurrent instances on a system’s CPU or GPU, running with FP32 precision instead of FP16, etc. The picture below shows the preconfigured files that are currently available for systems running Ubuntu on Intel hardware.

Because potential AIXPRT use cases cut across a wide range of hardware segments, including desktops, edge devices, and servers, not all AIXPRT workloads and configs will be applicable to each segment. As we move towards the AIXPRT GA, we’re working to find the best way to parse out these distinctions and communicate them to end users. In many cases, the ideal combination of test configuration variables remains an open question for ongoing research. However, we hope the alternative configuration files will help by giving testers a starting place.

If you experiment with an alternative test configuration file, please note that it should replace the existing default config file. If more than one config file is present, AIXPRT will run all the configurations and generate a separate result for each. More information about the config files and detailed instructions for how to handle the files are available in the EditConfig.md document in the public repository.

We’ll continue to keep everyone up to date with AIXPRT news here in the blog. If you have any questions or comments, please let us know.

Justin

Posted in AI, AIXPRT, benchmark, Benchmark metrics, Collaborative benchmark development, Community Preview, GitHub, Intel, Linux, Machine learning, NVIDIA, Performance benchmarking, Ubuntu | Tagged AI, AIXPRT, batch size, concurrent instances, FP16, FP32, GitHub, Intel, NVIDIA, precision, Windows |

AIXPRT Community Preview 3 is here!

By Justin Greene

on October 3, 2019

We’re happy to announce that the AIXPRT Community Preview 3 (CP3) is now available! As we discussed in last week’s blog, testers can expect three significant changes in AIXPRT CP3:

We updated support for the Ubuntu test packages from Ubuntu version 16.04 LTS to version 18.04 LTS.
We added TensorRT test packages for Windows and Ubuntu. Previously, AIXPRT testers could test only the TensorFlow variant of TensorRT. Now, they can use TensorRT to test systems with NVIDIA GPUs.
We added the Wide and Deep recommender system workload with the MXNet toolkit for Ubuntu systems.

To access AIXPRT CP3, click this access link and submit the brief information form unless you’ve already done so for CP2. You will then gain access to the AIXPRT community preview page. (If you’re not already a BenchmarkXPRT Development Community member, we’ll contact you with more information about your membership.)

On the community preview page, a download table displays the currently available AIXPRT CP3 test packages. Locate the operating system and toolkit you wish to test, and click the corresponding Download link. For detailed installation instructions and information on hardware and software requirements for each package, click the corresponding Readme link. Instead of providing installation guide PDFs as we did for CP2, we are now directing testers to a public GitHub repository. The repository contains the installation readmes for all the test packages, as well as a selection of alternative test configuration files. We’ll discuss the alternative configuration files in more detail in a future blog post.

Note: Those who have access to the existing AIXPRT GitHub repository will be able to access CP3 in the same way as previous versions.

We’ll continue to keep everyone up to date with AIXPRT news here in the blog. If you have any questions or comments, please let us know.

Justin

Posted in AI, AIXPRT, benchmark, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Future of performance evaluation, image classification, Machine learning, MXNet, NVIDIA, object detection, OpenVINO, Performance benchmarking, recommender system, ResNet-50, SSD-MobileNet v1, TensorFlow, TensorRT, Ubuntu, Wide and Deep | Tagged AI, AIXPRT, benchmark, image processing, machine learning, MXNet, object detection, OpenVINO, preview, recommender, TensorFlow, TensorRT, Ubuntu, Windows |

An update on AIXPRT development

By Justin Greene

on September 26, 2019

It’s been a while since we last discussed the AIXPRT Community Preview 3 (CP3) release schedule, so we want to let everyone know where things stand. Testing for CP3 has taken longer than we predicted, but we believe we’re nearly ready for the release.

Testers can expect three significant changes in AIXPRT CP3. First, we updated support for the Ubuntu test packages. During the initial development phase of AIXPRT, Ubuntu version 16.04 LTS (Long Term Support) was the most current LTS version, but version 18.04 is now available.

Second, we have added TensorRT test packages for Windows and Ubuntu. Previously, AIXPRT testers could test only the TensorFlow variant of TensorRT. Now, they can use TensorRT to test systems with NVIDIA GPUs.

Third, we have added the Wide and Deep recommender system workload with the MXNet toolkit. Recommender systems are AI-based information-filtering tools that learn from end user input and behavior patterns and try to present them with optimized outputs that suit their needs and preferences. If you’ve used Netflix, YouTube, or Amazon accounts, you’ve encountered recommender systems that learn from your behavior.

Currently, the recommender system workload in AIXPRT CP3 is available for Ubuntu testing, but not for Windows. Recommender system inference workloads typically run on datacenter hardware, which tends to be Linux based. If enough community members are interested in running the MXNet/Wide and Deep test package on Windows, we can investigate what that would entail. If you’d like to see that option, please let us know.

As always, if you have any questions about the AIXPRT development process, feel free to ask!

Justin

Posted in AI, AIXPRT, benchmark, Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Linux, Machine learning, MXNet, Performance benchmarking, Ubuntu, What makes a good benchmark?, Wide and Deep, Windows |

An updated HDXPRT 4 v1.1 installer package

By Justin Greene

on September 19, 2019

Today, we published an updated HDXPRT 4 v1.1 installer package that addresses an issue brought to light by HDXPRT testers and our own follow-up testing. We’ve also encountered an issue caused by anti-virus program interference during the HDXPRT installation process, so we’re providing steps for a workaround below. Neither the updated build nor the workaround steps affect the comparability of previous HDXPRT 4 test scores.

The first issue involves the hdxprt4.exe setup file. You may recall that the main updates in HDXPRT 4 v1.1 were the inclusion of the latest version of HandBrake and the ability for testers to choose whether to target a system’s discrete graphics card during the Convert Videos workload. Prior to today’s update, the HDXPRT 4 v1.1 installation package mistakenly included an old hdxprt4.exe setup file, which likely caused problems for testers attempting to target discrete graphics. We apologize for this oversight. The installer package we published today includes the correct hdxprt4.exe setup file.

The second issue is that during the installation process, Windows Security and other anti-virus programs may quarantine some of the AutoIt executables that HDXPRT 4 uses to install real-world applications, and the incomplete installation process will cause the test to fail. The files do not contain viruses, but the anti-virus programs may assume that the user has not granted HDXPRT permission to install the ancillary files. One of the executables currently triggering this behavior is the MediaEspresso ME75_2x4K_transcode.exe file. To check whether your test system is quarantining this file, navigate to the C:\Program Files (x86)\HDXPRT4\HDXPRT4_Workloads\HDXPRT4_Tests folder. Once the installation process is complete, the folder should contain 32 files, including ME75_2x4K_transcode.exe. If you see all 32 files, you’re ready to test. (Note: Once you run the test, HDXPRT 4 will add HDXPRTRunLog.txt to the folder, so you might see 33 files.)

If you see only 31 files, ME75_2x4K_transcode.exe is likely missing. To restore it, use the following steps:

1. Open the Windows Security app.
2. Select Virus & threat protection.
3. Under Current threats, select Protection history.
4. Check to see if Windows Security removed any threats around the time you installed HDXPRT 4.
5. If so, click the drop-down menu on the right side, where Windows Security lists the severity of the threat, and look for a false positive that reports the ME75_2x4K_transcode.exe file as Trojan:Win32/Wacatac.B!ml.
6. Click the Actions drop-down menu, and select Restore.
7. Navigate to the C:\Program Files (x86)\HDXPRT4\HDXPRT4_Workloads\HDXPRT4_Tests folder, and check to see where the ME75_2x4K_transcode.exe file is present.

Windows Security and other anti-virus programs may quarantine other HDXPRT installation files in the future. If your first HDXPRT 4 run fails to complete successfully, we suggest checking the anti-virus quarantine for HDXPRT-related files.

We also updated the HDXPRT 4 User Manual to include the steps above. If you have any questions about any of these topics, please feel free to contact us.

Justin

Posted in Application-based benchmarks, Benchmarking, HDXPRT, HDXPRT 4, HDXPRT workloads, Windows, Windows 10 | Tagged AutoIt, HDXPRT, HDXPRT 4, MediaEspresso, Windows, Windows Security |

Understanding concurrent instances in AIXPRT

By Justin Greene

on September 12, 2019

Over the past few weeks, we’ve discussed several of the key configuration variables in AIXPRT, such as batch size and level of precision. Today, we’re discussing another key variable: number of concurrent instances. In the context of machine learning inference, this refers to how many instances of the network model (ResNet-50, SSD-MobileNet, etc.) the benchmark runs simultaneously.

By default, the toolkits in AIXPRT run one instance at a time and distribute the compute load according to the characteristics of the CPU or GPU under test, as well as any relevant optimizations or accelerators in the toolkit’s reference library. By setting the number of concurrent instances to a number greater than one, a tester can use multiple CPUs or GPUs to run multiple instances of a model at the same time, usually to increase throughput.

With multiple concurrent instances, a tester can leverage additional compute resources to potentially achieve higher throughput without sacrificing latency goals.

In the current version of AIXPRT, testers can run multiple concurrent instances in the OpenVINO, TensorFlow, and TensorRT toolkits. When AIXPRT Community Preview 3 becomes available, this option will extend to the MXNet toolkit. OpenVINO and TensorRT automatically allocate hardware for each instance and don’t let users make manual adjustments. TensorFlow and MXNet require users to manually bind instances to specific hardware. (Manual hardware allocation for multiple instances is more complicated than we can cover today, so we may devote a future blog entry to that topic.)

Setting the number of concurrent instances in AIXPRT

The screenshot below shows part of a sample config file (the same one we used when we discussed batch size and precision). The value in the “concurrent instances” row indicates how many concurrent instances will be operating during the test. In this example, the number is one. To change that value, a tester simply replaces it with the desired number and saves the changes.

If you have any questions or comments (about concurrent instances or anything else), please feel free to contact us.

Justin

Posted in AI, AIXPRT, Benchmarking, Benchmarking computing devices, Community Preview, Cross-platform benchmarks, Machine learning, MXNet, OpenVINO, Performance benchmarking, ResNet-50, SSD-MobileNet v1, TensorFlow, TensorRT | Tagged AI, AIXPRT, batch size, concurrent instances, CPU, GPU, MXNet, OpenVINO, precision, TensorFlow, TensorRT |

Category: Benchmarking

A necessary update for HDXPRT 4

How to use alternate configuration files with AIXPRT

AIXPRT Community Preview 3 is here!

An update on AIXPRT development

An updated HDXPRT 4 v1.1 installer package

Understanding concurrent instances in AIXPRT

Setting the number of concurrent instances in AIXPRT

Check out the other XPRTs: