BenchmarkXPRT Blog banner

Tag Archives: data analytics

The CloudXPRT v1.2 update package is now available!

Justin Greene
By Justin Greene
on October 27, 2022

We’re happy to announce that the CloudXPRT v1.2 update package is now available! The update prevents potential installation failures on Google Cloud Platform and Microsoft Azure, and ensures that the web microservices workload works on Ubuntu 22.04. The update uses updated software components such as Kubernetes v1.23.7, Kubespray v2.18.1, and Kubernetes Metrics Server v1, and incorporates some additional minor script changes.

The CloudXPRT v1.2 web microservices workload installation package is available at the CloudXPRT.com download page and the BenchmarkXPRT GitHub repository.

Before you get started with v1.2, please note the following updated system requirements:

  • Ubuntu 20.04.2 or 22.04 for on-premises testing
  • Ubuntu 18.04, 20.04.2, or 22.04 for CSP (AWS/Azure/GCP) testing

Because CloudXPRT is designed to run on high-end servers, physical nodes or VMs under test must meet the following minimum specifications:

  • 16 logical or virtual CPUs
  • 8 GB of RAM
  • 10 GB of available disk space (50 GB for the data analytics workload)

The update package includes only the updated v1.2 test harness and the updated web microservices workload. It does not include the data analytics workload. As we stated in the blog, now that we’ve published the web microservices package, we will assess the level of interest users express about a possible refresh of the v1.1 data analytics workload. For now, the v1.1 data analytics workload will continue to be available via CloudXPRT.com for some time to serve as a reference resource for users who have worked with the package in the past.

Please let us know if you have any questions about the CloudXPRT v1.2 test package. Happy testing!

Justin

Posted in Amazon Web Services, benchmark, Benchmark metrics, Cloud, CloudXPRT, Cross-platform benchmarks, CSP, data analytics, Datacenter, Google Cloud Platform, Kubernetes, microservices, Microsoft Azure, On-premise, Performance benchmarking, Servers, Ubuntu, web microservices | Also tagged , , , , , , , , , , ,

Looking back on 2021 with the XPRTs

Justin Greene
By Justin Greene
on January 6, 2022

As 2022 gets underway, we want to take this opportunity to look back on 2021 and review another productive year for the XPRTs. Readers of our newsletter are familiar with the stats and updates we include each month, but for our blog readers who don’t receive the newsletter, we’ve compiled some highlights below.

Benchmarks
In the past year, we released the WebXPRT 4 Preview, CloudXPRT v1.1, and an updated CrXPRT 2 build that included a fix for prior issues with the battery life test.

XPRTs in the media
Journalists, advertisers, and analysts referenced the XPRTs thousands of times in 2021. It’s always rewarding to know that the XPRTs have proven to be useful and reliable assessment tools for technology publications such as AnandTech, Expert Reviews, Gadgets 360, Gizmodo, Hot Hardware, Laptop Mag, Legit Reviews, Notebookcheck, PCMag, PCWorld, TechPowerUp, Tom’s Hardware, and ZDNet.

Downloads and confirmed runs
In 2021, we had more than 23,600 benchmark downloads and 228,900 confirmed runs. Our most popular benchmark, WebXPRT, just passed 909,800 runs since its debut in 2013! WebXPRT continues to be a go-to, industry-standard performance benchmark for OEM labs, vendors, and leading tech press outlets around the globe.

Media, publications, and interactive tools
Part of our mission with the XPRTs is to produce tools and materials that help testers better understand the ins and outs of benchmarking in general and the XPRTs in particular. To help achieve this goal, we published the following in 2021:

We’re thankful for everyone who has used the XPRTs, joined the community, and sent questions and suggestions throughout 2021. We look forward to an exciting 2022!

Justin

Posted in Battery life, battery life, benchmark, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, CloudXPRT, Collaborative benchmark development, CrXPRT, data analytics, results, WebXPRT, webxprt 4 | Also tagged , , , , , ,

We welcome your CloudXPRT results!

Justin Greene
By Justin Greene
on July 15, 2021

We recently published a set of CloudXPRT Data Analytics and Web Microservices workload test results submitted by Quanta Computer, Inc. The Quanta submission is the first set of CloudXPRT results that we’ve published using the formal results submission and approval process. We’re grateful to the Quanta team for carefully following the submission guidelines, enabling us to complete the review process without a hitch.

If you are unfamiliar with the process, you can find general information about how we review submissions in a previous blog post. Detailed, step-by-step instructions are available on the results submission page. As a reminder for testers who are considering submitting results for July, the submission deadline is tomorrow, Friday July 16, and the publication date is Friday July 30. We list the submission and publication dates for the rest of 2021 below. Please note that we do not plan to review submissions in December, so if we receive results submissions after November 30, we may not publish them until the end of January 2022.

August

Submission deadline: Tuesday 8/17/21

Publication date: Tuesday 8/31/21

September

Submission deadline: Thursday 9/16/21

Publication date: Thursday 9/30/21

October

Submission deadline: Friday 10/15/21

Publication date: Friday 10/29/21

November

Submission deadline: Tuesday 11/16/21

Publication date: Tuesday 11/30/21

December

Submission deadline: N/A

Publication date: N/A

If you have any questions about the CloudXPRT results submission, review, or publication process, please let us know!

Justin

Posted in BenchmarkXPRT, Cloud, CloudXPRT, Cross-platform benchmarks, data analytics, Datacenter, microservices, Performance benchmarking, results, results submission, Server, Servers, web microservices | Also tagged , , , , , , ,

Default requirements for CloudXPRT results submissions

Justin Greene
By Justin Greene
on April 22, 2021

Over the past few weeks, we’ve received questions about whether we require specific test configuration settings for official CloudXPRT results submissions. Currently, testers have the option to edit up to 12 configuration options for the web microservices workload and three configuration options for the data analytics workload. Not all configuration options have an impact on testing and results, but a few of them can drastically affect key results metrics and how long it takes to complete a test. Because new CloudXPRT testers may not anticipate those outcomes, and so many configuration permutations are possible, we’ve come up with a set of requirements for all future results submissions to our site. Please note that testers are still free to adjust all available configuration options—and define service level agreement (SLA) settings—as they see fit for their own purposes. The requirements below apply only to results testers want to submit for publication consideration on our site, and to any resulting comparisons.


Web microservices results submission requirement

Starting with the May results submission cycle, all web microservices results submissions must have the workload.cpurequestsvalue, which lets the user designate the number of CPU cores the workload assigns to each pod, set to 4. Currently, the benchmark supports values of 1, 2, and 4, with the default value of 4. While 1 and 2 CPU cores per pod may be more appropriate for relatively low-end systems or configurations with few vCPUs, a value of 4 is appropriate for most datacenter processors, and it often enables CSP instances to operate within the benchmark’s max default 95th percentile latency SLA of 3,000 milliseconds.

In future CloudXPRT releases, we may remove the option to change the workload.cpurequests value from the config.json file and simply fix the value in the benchmark’s code to promote test predictability and reasonable comparisons. For more information about configuration options for the web microservices workload, please consult the Overview of the CloudXPRT Web Microservices Workload white paper.


Data analytics results submission requirement

Starting with the May results submission cycle, all data analytics results submissions must have the best reported performance (throughput_jobs/min) correspond to a 95th percentile SLA latency of 90 seconds or less. We have received submissions where the throughput was extremely high, but the 95th percentile SLA latency was up to 10 times the 90 seconds that we recommend in CloudXPRT documentation. High latency values may be acceptable for the unique purposes of individual testers, but they do not provide a good basis for comparison between clusters under test. For more information about configuration options with the data analytics workload, please consult the Overview of the CloudXPRT Data Analytics Workload white paper.

We will update CloudXPRT documentation to make sure that testers know to use the default configuration settings if they plan to submit results for publication. If you have any questions about CloudXPRT or the CloudXPRT results submission process, please let us know.

Justin

Posted in BenchmarkXPRT, Cloud, CloudXPRT, Clusters, data analytics, Datacenter, IaaS, microservices, Performance benchmarking, results, results submission, Servers, web microservices, White papers | Also tagged , , , , , , ,

The CloudXPRT v1.1 beta is available!

Justin Greene
By Justin Greene
on February 18, 2021

Last week, we announced that a CloudXPRT v1.1 beta was on the way. We’re happy to say that the v1.1 beta is now available to the public on a dedicated CloudXPRT v1.1 beta download page. While CloudXPRT v1.01 remains the officially supported version on CloudXPRT.com and in our GitHub repository, interested testers can use the v1.1 beta version in new environments as we finalize the v1.1 build for official release. You are welcome to publish results as we do not expect results to change in the final, official release.

As we mentioned in last week’s post, the CloudXPRT v1.1 beta includes the following changes:

  • We’ve added support for Ubuntu 20.04.2 or later for on-premises testing.
  • We’ve consolidated and standardized the installation packages for both workloads. Instead of one package for the data analytics workload and four separate packages for the web microservices workload, each workload has a single installation package that supports on-premises testing and testing with all three supported CSPs.
  • We’ve incorporated Terraform to help create and configure VMs, which helps to prevent problems when testers do not allocate enough storage per VM prior to testing.
  • We’ve replaced the Calico network plugin in Kubespray with Weave, which helps to avoid some of the network issues testers have occasionally encountered in the CPS environment.

Please feel free to share the link to the beta download page. (To avoid confusion, the beta will not appear in the main CloudXPRT download table.) We can’t yet state definitively whether results from the new version will be comparable to those from v1.01. We have not observed any significant differences in performance, but we haven’t tested every possible test configuration across every platform. If you observe different results when testing the same configuration with v1.01 and v1.1 beta, please send us the details so we can investigate.

If you have any questions about CloudXPRT or the CloudXPRT v1.1 beta, please let us know!

Justin

Posted in benchmark, Benchmarking, BenchmarkXPRT, Cloud, CloudXPRT, Cross-platform benchmarks, data analytics, Datacenter, Hosted cloud, Hybrid cloud, IaaS, Kubernetes, microservices, on premises, Performance benchmarking, Performance of computing devices, results, Server, Ubuntu, web microservices | Also tagged , , , , , , , , ,

The CloudXPRT v1.1 beta is on the way

Justin Greene
By Justin Greene
on February 11, 2021

As we’ve been working on improvements and updates for CloudXPRT, we’ve been using feedback from community members to determine which changes will help testers most in the short term. To make some of those changes available to the community as soon as possible, we plan to release a beta version of CloudXPRT v1.1 in the coming weeks.

During the v1.1 beta period, the CloudXPRT v1.01 installation packages on CloudXPRT.com and our GitHub repository will continue to include the officially supported version of CloudXPRT. However, interested testers can experiment with the v1.1 beta version in new environments while we finalize the build for official release. 

The CloudXPRT v1.1 beta includes the following primary changes:

  • We’re adding support for Ubuntu 20.04.2 or later, the number one request we’ve received.
  • We’re consolidating and standardizing the installation packages for both workloads. Instead of one package for the data analytics workload and four separate packages for the web microservices workload, each workload will have two installation packages: one for all on-premises testing and one for testing with all three supported CSPs.
  • We’re incorporating Terraform to help create and configure VMs, which will help to prevent situations when testers do not allocate enough storage per VM prior to testing.
  • We use Kubespray to manage Kubernetes clusters, and Kubespray uses Calico as the default network plug in. Calico has not always worked well for CloudXPRT in the CSP environment, so we’re replacing Calico with Weave.


At the start of the beta period, we will share a link to the v1.1 beta download page here in the blog. You’ll be free to share this link. To avoid confusion, we will not add the beta download to the v1.01 downloads available on CloudXPRT.com.

As the beta release date approaches, we’ll share more details about timelines, access, and any additional changes to the benchmark. If you have any questions about the upcoming CloudXPRT v1.1 beta, please let us know!

Justin

Posted in benchmark, Benchmark metrics, BenchmarkXPRT, BenchmarkXPRT development community, Cloud, CloudXPRT, Clusters, Collaborative benchmark development, Cross-platform benchmarks, data analytics, Datacenter, GitHub, Hosted cloud, IaaS, Kubernetes, Linux, on premises, Performance benchmarking, web microservices | Also tagged , , , , , , , ,

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2024   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x