Tag Archives: NVIDIA

How to use alternate configuration files with AIXPRT

on October 10, 2019

In last week’s AIXPRT Community Preview 3 announcement, we mentioned the new public GitHub repository that we’re using to publish AIXPRT-related information and resources. In addition to the installation readmes for each AIXPRT installation package, the repository contains a selection of alternative test config files that testers can use to quickly and easily change a test’s parameters.

As we discussed in previous blog entries about batch size, levels of precision, and number of concurrent instances, AIXPRT testers can adjust each of these key variables by editing the JSON file in the AIXPRT/Config directory. While the process is straightforward, editing each of the variables in a config file can take some time, and testers don’t always know the appropriate values for their system. To address both of these issues, we are offering a selection of alternative config files that testers can download and drop into the AIXPRT/Config directory.

In the GitHub repository, we’ve organized the available config files first by operating system (Linux_Ubuntu and Windows) and then by vendor (All, Intel, and NVIDIA). Within each section, testers will find preconfigured JSON files set up for several scenarios, such as running with multiple concurrent instances on a system’s CPU or GPU, running with FP32 precision instead of FP16, etc. The picture below shows the preconfigured files that are currently available for systems running Ubuntu on Intel hardware.

Because potential AIXPRT use cases cut across a wide range of hardware segments, including desktops, edge devices, and servers, not all AIXPRT workloads and configs will be applicable to each segment. As we move towards the AIXPRT GA, we’re working to find the best way to parse out these distinctions and communicate them to end users. In many cases, the ideal combination of test configuration variables remains an open question for ongoing research. However, we hope the alternative configuration files will help by giving testers a starting place.

If you experiment with an alternative test configuration file, please note that it should replace the existing default config file. If more than one config file is present, AIXPRT will run all the configurations and generate a separate result for each. More information about the config files and detailed instructions for how to handle the files are available in the EditConfig.md document in the public repository.

We’ll continue to keep everyone up to date with AIXPRT news here in the blog. If you have any questions or comments, please let us know.

Justin

Posted in AI, AIXPRT, benchmark, Benchmark metrics, Collaborative benchmark development, Community Preview, GitHub, Intel, Linux, Machine learning, NVIDIA, Performance benchmarking, Ubuntu | Also tagged AI, AIXPRT, batch size, concurrent instances, FP16, FP32, GitHub, Intel, precision, Windows

A new HDXPRT 4 build is available!

By Justin Greene

on July 18, 2019

A few weeks ago, we announced that a new HDXPRT 4 build, v1.1, was on the way. This past Monday, we published the build on HDXPRT.com.

The new build includes an updated version of HandBrake, the commercial application that HDXPRT uses for certain video conversion tasks. HandBrake 1.2.2 supports hardware acceleration with AMD Video Coding Engine (VCE), Intel Quick Sync, and the NVIDIA video encoder (NVENC). By default, HDXPRT4 v1.1 uses the encoder available through a system’s integrated graphics, but testers can target discrete graphics by changing a configuration file flag before running the benchmark. HDXPRT will then use the encoder provided by the discrete graphics hardware. This configuration setting takes effect only when more than one of the supported encoders (VCE, QSV, or NVENC) is present on the system.

As we mentioned before, in all other respects, the benchmark has not changed. That means that, apart from a scenario where a tester changes the targeted graphics hardware, scores from previous HDXPRT 4 builds will be comparable to those from the new build.

The updated HDXPRT 4 User Manual contains additional information and instructions for changing the configuration file flag. Please contact us if you have any questions about the new build. Happy testing!

Justin

Posted in AMD, benchmark, BenchmarkXPRT development community, graphics, HDXPRT, HDXPRT 4, NVIDIA, Performance benchmarking, Windows 10 | Also tagged AMD, graphics, HDXPRT, HDXPRT 4, Intel, NVENC, QSV, VCE, Windows

An updated HDXPRT 4 build is on the way

By Justin Greene

on July 3, 2019

HandBrake recently released a new version, v1.2.2, of their video conversion software. Among other improvements, the new version includes support for certain AMD (VCE) and NVIDIA (NVENC) hardware-accelerated video encoders. Because we include HandBrake as one of the commercial applications in the HDXPRT installer package, and because we want to keep HDXPRT 4 up-to-date for testers, we’ve put together a new HDXPRT 4 build: v1.1. It includes HandBrake 1.2.2’s new capabilities, and we’re currently testing it in the lab.

With the new build, testers will be able to choose whether HDXPRT’s HandBrake tasks target a system’s integrated or discrete graphics cards by changing a flag called “UseIntegrated” in the config file. In HDXPRT 4 v1.1, the flag is set to “true” by default, directing HandBrake to use the codec provided by the system’s integrated graphics hardware. On the other hand, if a system has both integrated and discrete graphics available, and a user sets the flag to “false,” HandBrake will use the codec provided by the discrete graphics.

This update allows users to compare the video conversion performance of different video codecs on the same system. In all other respects, the benchmark has not changed. So apart from a scenario where a tester changes the targeted graphics hardware, scores from previous HDXPRT 4 builds will be comparable to those from the new build.

We’ll let the community know as soon as the new build is available, and we’ll update the HDXPRT 4 User Manual to reflect the changes.

If you have any questions about the upcoming HDXPRT 4 build, please let us know!

Justin

Posted in AMD, Future of performance evaluation, HandBrake, HDXPRT, HDXPRT 4, HDXPRT capabilities, HDXPRT release cycle, NVIDIA, Performance benchmarking | Also tagged AMD, codec, discrete, Handbrake, HDXPRT, HDXPRT 4, integrated, video

Preparing for the AIXPRT Community Preview

By Bill Catchings

on February 21, 2019

Thanks to everyone who downloaded the AIXPRT Request for Comments (RFC) preview build. Next week, we’re planning to publish the AIXPRT Community Preview (CP). The AIXPRT CP build includes support for the Intel OpenVINO, TensorFlow (CPU and GPU), and TensorFlow with NVIDIA TensorRT toolkits to run image-classification workloads with ResNet-50 and SSD-MobileNet v1 networks. The test reports FP32, FP16, and INT8 levels of precision. As with the RFC build, the test systems must be running Ubuntu 16.04 LTS. The minimum CPU and GPU requirements vary according to the toolkit being used, and we will publish more details about the hardware minimums next week.

As with our other community previews, we think the AIXPRT CP candidate is solid enough to allow folks to start quoting test results. During CP periods, we generally allow members to publish their own results, but wait until the build is available to the public before we post results on our site. Because community feedback is especially important for AIXPRT, we will handle things a bit differently. During the CP period, we’ll publish results that we produce as well as those from other members of the community, which you’ll be able to view at AIXPRT.com.

We’ll also provide detailed instructions for publishing results and sending them to us. Because of the high number of variables in each potential test configuration, we’ll ask testers to disclose more test, software, and hardware information than in the past. We will make this information available along with the results on AIXPRT.com. Our goal is that others can reproduce these numbers and confirm that they get similar results.

Our CP periods typically last four to six weeks before we make the benchmark available to the general public. If that schedule holds, it would place the public AIXPRT release around the end of March. During the CP period, we welcome your thoughts and suggestions about all aspects of the benchmark.

Also, we normally restrict access to our CPs to BenchmarkXPRT Development Community members. However, because we’re seeking broad input from experts in this field, we’ll gladly make the CP available to anyone interested in participating who has a GitHub account. To gain access, please contact us and let us know your GitHub username. Once we receive it, we’ll send you an invitation to join the repository as a collaborator.

Please let us know if you have any questions. We look forward to hearing your feedback.

Bill

Posted in Uncategorized | Also tagged AI, AIXPRT, BenchmarkXPRT, BenchmarkXPRT Development Community, Community Preview, GitHub, Intel, machine learning, MobileNet, OpenVINO, ResNet, TensorFlow, TensorRT, test

Tag Archives: NVIDIA

How to use alternate configuration files with AIXPRT

A new HDXPRT 4 build is available!

An updated HDXPRT 4 build is on the way

Preparing for the AIXPRT Community Preview

Check out the other XPRTs: