BenchmarkXPRT Blog banner

Category: Hybrid cloud

On track for a CloudXPRT web microservices update this fall

Justin Greene
By Justin Greene
on September 29, 2022

Last month, we announced that we’re working on an updated CloudXPRT web microservices test package. The purpose of the update is to fix installation failures on Google Cloud Platform and Microsoft Azure, and ensure that the web microservices workload works on Ubuntu 22.04, using updated software components such as Kubernetes v1.23.7, Kubespray v2.18.1, and Kubernetes Metrics Server v1. The update also incorporates some additional minor script changes.

We are still testing the updated test package with on-premises hardware and Amazon Web Services, Google Cloud Platform, and Microsoft Azure configurations. So far, testing is progressing well, and we feel increasingly confident that we will be able to release the updated test package soon. We would like to share a more concrete release schedule, but because of the complexity of the workload and the CSP platforms involved, we are waiting until we are certain that everything is ready to go.

The name of the updated package will be CloudXPRT v1.2, and it will include only the updated v1.2 test harness and the updated web microservices workload. It will not include the data analytics workload. As we stated in last month’s blog, we plan to publish the updated web microservices package, and see what kind of interest we receive from users about a possible refresh of the v1.1 data analytics workload. For now, the v1.1 data analytics workload will continue to be available via CloudXPRT.com for some time to serve as a reference resource for users that have worked with the package in the past.

As soon as possible, we’ll provide more information about the CloudXPRT v1.2 release date here in the blog. If you have any questions about the update or CloudXPRT in general, please feel free to contact us!

Justin

Posted in Amazon Web Services, benchmark, Cloud, CloudXPRT, Cross-platform benchmarks, Google Cloud Platform, Hosted cloud, Hybrid cloud, IaaS, Kubernetes, Microsoft Azure, on premises, Performance benchmarking | Tagged , , , , , , , , , |

CloudXPRT status and next steps

Justin Greene
By Justin Greene
on August 11, 2022

We developed our first cloud benchmark, CloudXPRT, to measure the performance of cloud applications deployed on modern infrastructure as a service (IaaS) platforms. When we first released CloudXPRT in February of 2021, the benchmark included two test packages: a web microservices workload and a data analytics workload. Both supported on-premises and cloud service provider (CSP) testing with Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. 

CloudXPRT is our most complex benchmark, requiring sustained compatibility between many software components across multiple independent test environments. As vendors roll out updates for some components and stop supporting others, it’s inevitable that something will break. Since CloudXPRT’s launch, we’ve become aware of installation failures while attempting to set up CloudXPRT on Ubuntu virtual machines with GCP and Microsoft Azure. Additionally, while the web microservices workload continues to run in most instances with a few configuration tweaks and workarounds, the data analytics workload fails consistently due to compatibility issues with Minio, Prometheus, and Kafka within the Kubernetes environment. 

In response, we’re working to fix problems with the web microservices workload and bring all necessary components up to date. We’re developing an updated test package that will work on Ubuntu 22.04, using Kubernetes v1.23.7 and Kubespray v2.18.1. We’re also updating Kubernetes Metrics Server from v1beta1 to v1, and will incorporate some minor script changes. Our goal is to ensure successful installation and testing with the on-premises and CSP platforms that we supported when we first launched CloudXPRT.

We are currently focusing on the web microservices workload for two reasons. First, more users have downloaded it than the data analytics workload. Second, we think we have a clear path to success. Our plan is to publish the updated web microservices test package, and see what feedback and interest we receive from users about a possible data analytics refresh. The existing data analytics workload will remain available via CloudXPRT.com for the time being to serve as a reference resource.

We apologize for the inconvenience that these issues have caused. We’ll provide more information about a release timeline and final test package details here in the blog as we get closer to publication. If you have any questions about the future of CloudXPRT, please feel free to contact us!

Justin

Posted in Amazon Web Services, benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, Cloud, CloudXPRT, Cross-platform benchmarks, CSP, data analytics, Datacenter, Google Cloud Platform, Hosted cloud, Hybrid cloud, IaaS, Kubernetes, microservices, Microsoft Azure, On-premise, Performance benchmarking, web microservices | Tagged , , , , , , , , , , |

The CloudXPRT v1.1 general release is tomorrow!

Justin Greene
By Justin Greene
on March 11, 2021

We’re happy to announce that CloudXPRT v1.1 will move from beta to general release status tomorrow! The installation packages will be available at the CloudXPRT.com download page and the BenchmarkXPRT GitHub repository. You will find more details about the v1.1 updates in a previous blog post, but the most prominent changes are the consolidation of the five previous installation packages into two packages (one per workload) and added support for Ubuntu 20.04.2 with on-premises testing.

Before you get started with v1.1, please note the following updated system requirements:

  • Ubuntu 20.04.2 or later for on-premises testing
  • Ubuntu 18.04 and 20.04.2 or later for CSP (AWS/Azure/GCP) testing

CloudXPRT is designed to run on high-end servers. Physical nodes or VMs under test must meet the following minimum specifications:

  • 16 logical or virtual CPUs
  • 8 GB of RAM
  • 10 GB of available disk space (50 GB for the data analytics workload)

We have also made significant adjustments to the installation and test configuration instructions in the readmes for both workloads, so please revisit these documents even if you’re familiar with previous test processes.

As we noted during the beta period, we have not observed any significant differences in performance between v1.01 and v1.1, but we haven’t tested every possible test configuration across every platform. If you observe different results when testing the same configuration with v1.01 and v1.1, please send us the details so we can investigate.

If you have any questions about CloudXPRT v1.1, please let us know!

Justin

Posted in Amazon Web Services, benchmark, Benchmarking, BenchmarkXPRT, Cloud, CloudXPRT, Clusters, Cross-platform benchmarks, CSP, data analytics, Datacenter, Google Cloud Platform, Hosted cloud, Hybrid cloud, IaaS, Microsoft Azure, on premises, Performance benchmarking, Server, Ubuntu, web microservices | Tagged , , , , , , , , |

The CloudXPRT v1.1 beta is available!

Justin Greene
By Justin Greene
on February 18, 2021

Last week, we announced that a CloudXPRT v1.1 beta was on the way. We’re happy to say that the v1.1 beta is now available to the public on a dedicated CloudXPRT v1.1 beta download page. While CloudXPRT v1.01 remains the officially supported version on CloudXPRT.com and in our GitHub repository, interested testers can use the v1.1 beta version in new environments as we finalize the v1.1 build for official release. You are welcome to publish results as we do not expect results to change in the final, official release.

As we mentioned in last week’s post, the CloudXPRT v1.1 beta includes the following changes:

  • We’ve added support for Ubuntu 20.04.2 or later for on-premises testing.
  • We’ve consolidated and standardized the installation packages for both workloads. Instead of one package for the data analytics workload and four separate packages for the web microservices workload, each workload has a single installation package that supports on-premises testing and testing with all three supported CSPs.
  • We’ve incorporated Terraform to help create and configure VMs, which helps to prevent problems when testers do not allocate enough storage per VM prior to testing.
  • We’ve replaced the Calico network plugin in Kubespray with Weave, which helps to avoid some of the network issues testers have occasionally encountered in the CPS environment.

Please feel free to share the link to the beta download page. (To avoid confusion, the beta will not appear in the main CloudXPRT download table.) We can’t yet state definitively whether results from the new version will be comparable to those from v1.01. We have not observed any significant differences in performance, but we haven’t tested every possible test configuration across every platform. If you observe different results when testing the same configuration with v1.01 and v1.1 beta, please send us the details so we can investigate.

If you have any questions about CloudXPRT or the CloudXPRT v1.1 beta, please let us know!

Justin

Posted in benchmark, Benchmarking, BenchmarkXPRT, Cloud, CloudXPRT, Cross-platform benchmarks, data analytics, Datacenter, Hosted cloud, Hybrid cloud, IaaS, Kubernetes, microservices, on premises, Performance benchmarking, Performance of computing devices, results, Server, Ubuntu, web microservices | Tagged , , , , , , , , , , |

We’ve fixed an installation bug in the CloudXPRT Data Analytics Workload package

Justin Greene
By Justin Greene
on February 4, 2021

Yesterday, we published an updated CloudXPRT Data Analytics workload package that fixes a problem during the package installation process. CloudXPRT uses the Helm utility, which serves as a package manager for the Kubernetes container orchestration system. Helm accesses files in a default repository, and the version of Helm that we originally used with CloudXPRT tries to access files that are no longer available. We fixed the problem by updating the code to use the latest version of Helm.

This update does not change how the benchmark workload runs, and has no impact on benchmark results. We apologize if this bug caused headaches for any testers during installation, and we appreciate your patience as we worked on a fix.

As a reminder for testers interested in experimenting with the CloudXPRT Data Analytics workload, the Overview of the CloudXPRT Data Analytics Workload paper is now available. You can find links to the paper and other resources in the Helpful Info box on CloudXPRT.com and the CloudXPRT section of our XPRT white papers page.

If you have any questions, or have encountered any obstacles during testing, please let us know!

Justin

Posted in benchmark, Benchmark metrics, BenchmarkXPRT, Cloud, CloudXPRT, Cross-platform benchmarks, data analytics, Datacenter, Hosted cloud, Hybrid cloud, Kubernetes, on premises, Performance benchmarking, White papers |

The Overview of the CloudXPRT Data Analytics Workload white paper is now available!

Justin Greene
By Justin Greene
on January 28, 2021

Today, we expand our portfolio of CloudXPRT resources with a paper on the benchmark’s data analytics workload. While we summarized the workload in the Introduction to CloudXPRT white paper, the new paper goes into much greater detail.

In addition to providing practical information about the data analytics installation package and minimum system requirements, the paper describes the workload’s test configuration variables, structural components, task workflows, and test metrics. It also discusses interpreting test results and the process for submitting results for publication.

CloudXPRT is the most complex tool in the XPRT family, and the new paper is part of our effort to create more—and better—CloudXPRT documentation. We plan to publish additional CloudXPRT white papers in the coming months, with possible future topics including the impact of adjusting specific test configuration options, recommendations for results reporting, and methods for analysis.

We hope that the Overview of the CloudXPRT Data Analytics Workload paper will serve as a go-to resource for CloudXPRT testers, and will answer any questions you have about the workload. You can find links to the paper and other resources in the Helpful Info box on CloudXPRT.com and the CloudXPRT section of our XPRT white papers page.

If you have any questions, please let us know!

Justin

Posted in Benchmark metrics, Benchmarking, BenchmarkXPRT, Cloud, CloudXPRT, Cross-platform benchmarks, data analytics, Datacenter, Hosted cloud, Hybrid cloud, on premises, Performance benchmarking, results, Servers, White papers | Tagged , , , |

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2024   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x