On premises (local data center)

On premises clouds reside on an organization’s local server/data center infrastructure, and are managed by the organization’s IT personnel.

Off premises (Amazon Web Services)

Amazon Web Services (AWS) is a cloud services provider that offers a variety of cloud, infrastructure as a service (IaaS), and platform as a service (PaaS) products through subscription and pay-as-you-go pricing.

Learn more about AWS at https://aws.amazon.com/.

Off premises (Google Cloud Platform)

Google Cloud Platform (GCP) is a suite of cloud, infrastructure as a service (IaaS), and platform as a service (PaaS) products that run on Google’s own datacenter infrastructure. GCP offers pay-as-you-go pricing.

Learn more at https://cloud.google.com/gcp.

Off premises (Microsoft Azure)

Microsoft Azure is a cloud computing service that provides a variety of cloud, infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS) products. Microsoft Azure offers pay-as-you-go pricing.

Learn more about Microsoft Azure at https://azure.microsoft.com/.

Hybrid clouds

We hope to add instructions for CloudXPRT testing on hybrid clouds in the future. Hybrid clouds are cloud infrastructure systems that comprise a mixture of on-premises, private cloud, and/or public cloud components. Businesses can use hybrid cloud solutions to optimize the balance of factors such as scalability, flexibility, on-demand performance, security, and cost.

Kubernetes

Testing for each of the four target platforms—on-premises, AWS, Azure, and GCP—requires Kubernetes. Kubernetes is an open-source, automated system for deploying and managing containers (virtualized, isolated computing spaces).

https://kubernetes.io/

Dockers

Testing for three of the four target platforms—on-premises, AWS, and GCP—requires Docker in addition to Kubernetes. Docker offers a set of open-source container deployment and management tools that allow a single operating system kernel to run multiple containers. We also use Docker to create the container images used in the benchmark. We host CloudXPRT images at the Docker Hub repository.

https://www.docker.com/

CloudXPRT currently supports two workloads: web microservices and data analytics. Testers can use metrics from the workloads to compare IaaS stack (both hardware and software) performance and to evaluate whether any given stack is capable of meeting service-level agreement (SLA) thresholds. Testers can run CloudXPRT on local data center, Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure deployments.

Web microservices

In the web microservices workload, a simulated user logs into a web application that does three things: provides a selection of stock options, performs Monte-Carlo simulations with those stocks, and presents the user with options that may be of interest. This scenario enables the workload to model a traditional three-tier web application with services in the web, application, and data layers. The workload uses Kubernetes, Docker, NGNIX, REDIS, Cassandra, and monitoring modules to create an end-to-end IaaS scenario. The workload reports performance in transactions per second, which reflects the number of successful requests per second the stack achieves for each level of concurrency. Testers can use this workload’s metrics to compare IaaS stack performance and to evaluate whether any given stack is capable of meeting SLA thresholds.

Data analytics

The data analytics workload calculates XGBoost model training time. XGBoost is a gradient-boosting framework that data scientists often use for machine learning (ML)-based regression and classification problems. The purpose of the workload in the context of CloudXPRT is to measure how well an IaaS stack enables XGBoost to speed up model training. To do this, the data analytics workload uses Kubernetes, Docker, object storage, message pipeline, and monitoring modules to create an end-to-end IaaS scenario. The workload reports latency in the form of job duration in seconds, and throughput rate in the form of jobs per minute. Testers can use these metrics to compare IaaS stack performance and to evaluate whether any given stack is capable of meeting SLA thresholds.

On what types of systems can I run CloudXPRT?

We designed CloudXPRT for high-end servers. During each workload, the benchmark scales to utilize all the available cores. However, for functional testing, the physical node(s) or VM(s) under test must have the following (at a minimum):

• Ubuntu 18.04 LTS
• 16 logical or virtual CPUs
• 8 GB RAM
• 10 GB of available disk space (20 GB for the data analytics workload)
• An internet connection

Supported operating systems

CloudXPRT runs on Ubuntu 18.04 LTS.

Minimum hardware requirements

We designed CloudXPRT for high-end servers. During each workload, the benchmark scales to utilize all the available cores. However, for functional testing, the physical node(s) or VM(s) under test must have the following (at a minimum):

• Ubuntu 18.04 LTS
• 16 logical or virtual CPUs
• 8 GB RAM
• 10 GB of available disk space (20 GB for the data analytics workload)
• An internet connection

For three of the four target platforms—on-premises, AWS, and GCP—testing requires Docker in addition to Kubernetes. Testing on Microsoft Azure requires only Kubernetes. Off-premises tests require access to an AWS, Azure, or GCP account, depending on the configuration you want to test.

Skills you’ll need

CloudXPRT testers need basic familiarity with the Ubuntu operating environment, as well as Kubernetes and Docker. Depending on the target IaaS platform, testers also need to be able to set up and configure virtual machines and clusters on local hardware or with Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure.

How long it takes

The length of a test run depends on the system under test and the test configuration settings. Test length can range from a few minutes to several hours.

Choosing a CloudXPRT download package

Testers can directly access all CloudXPRT test packages in the CloudXPRT download table. To simplify installation, we separated the CloudXPRT installation packages by workload and target platform. There are five installation packages: a single data analytics installation package, which is compatible with all four target platforms (on-premises, AWS, Azure, and GCP), and four web microservices installation packages, one for each of the four target platforms.

Configuring the test

Both CloudXPRT workloads can run using default test configuration values, but users have the option to adjust multiple test configuration variables for each workload. It is important to note that server/data center testing can often require extensive tuning to achieve maximum performance, so the ideal combination of configuration settings will vary according to each tester’s goals.

For more detailed information about configuring the CloudXPRT web microservices workload, read the Configure benchmark parameters section under the Run the benchmark on premises section of the CloudXPRT web microservices readme.

For more detailed information about configuring the CloudXPRT data analytics workload, read the Run the benchmark section of the CloudXPRT web microservices readme.

Metrics

Both CloudXPRT workloads report several primary and secondary metrics. Testers can use one or both sets of primary metrics to compare IaaS stack performance and evaluate whether any given stack is capable of meeting necessary service-level agreement (SLA) thresholds.

The CloudXPRT web microservices workload measures performance using a primary metric of transactions per second, which reflects the number of successful requests per second the stack achieves for each level of concurrency.

The CloudXPRT data analytics workload measures performance using the primary metrics of throughput rate in the form of jobs per minute, and latency in the form of job duration in seconds.

Why does this matter for servers and cloud instances?

Existing data center benchmarks don’t make it easy to understand how applications will perform on a given IaaS infrastructure. CloudXPRT uses cloud-native components on actual IaaS hardware and software stacks to provide end-to-end performance metrics that allow users to choose the best configuration for their business. Testers can use metrics from the CloudXPRT web microservices and data analytics workloads to compare the IaaS performance of different cloud service providers—or different server and cluster configurations with the same provider—and evaluate whether any given stack is capable of meeting necessary SLA thresholds.

At CloudXPRT.com, interested parties can view test results published by the BenchmarkXPRT Development Community. The following tips can help visitors navigate the results viewer:

• You can click the tabs at the top of the table to switch from web microservices results to data analytics results.
• You can click the header of any column to sort the data on that variable. One click sorts A-Z (smallest to largest for numeric values) and two clicks sort Z-A (largest to smallest for numeric values).
• You can click the link in the Source/details column to visit a detailed page on that result. The page contains additional test configuration and system hardware information and lets you download results files.
• You can search for variables such as cloud service or cluster components.
• The viewer displays eight results per page by default. You can change this to 16, 48, or Show all.

Publishing results yourself

You can publish your test results at any time. However, if you’d like to publish your results in the official CloudXPRT results table, you must submit your results to the review group by the results submission deadline each month. To do so, please follow the instructions on the CloudXPRT results submission page.

The results review group

The CloudXPRT results review group serves as a sanity check and a forum for comments on each month’s submissions. All registered BenchmarkXPRT Development Community members who wish to participate in the review process can join the group by contacting us via email. We’ll confirm receipt of your request and add you to the review group mailing list. Any non-members who would like to join the review group can contact us and we’ll help you become community members.

The results submission, review, and publication cycle

We update the CloudXPRT results database once a month on a published schedule. While testers can submit results through the CloudXPRT results submission page at any time, two weeks prior to each publication date, we close submissions for that review cycle. One week prior to each publication date, we email details of that month’s submissions to the results review group, along with the deadline for sending post-publication feedback.

The results publication schedule

We publish results to the database on the last business day of each month and close the submission window at 23:59Eastern Time (UTC-5) on the business day that falls two weeks earlier (with occasional adjustments for holidays). The schedule is available at least six months in advance on CloudXPRT.com.

Cloud computing

Cloud computing is the provision of a variety of data center-based computing services to multiple users over a network. Cloud services may include on-demand storage, networking, application hosting, and processing power, among many others.

On-premises clouds reside on an organization’s local server/data center infrastructure. Off-premises (publicly hosted) clouds reside on external infrastructure owned by a third party. Cloud services infrastructure may be physically isolated (private cloud), logically isolated (virtual private cloud), or share resources with other organizations (public cloud).

Learn more at https://en.wikipedia.org/wiki/Cloud_computing.

Cloud native

Cloud-native components and applications consist of sets of microservices designed for the dynamic virtualized environments of private, public, and hybrid clouds. CloudXPRT uses Docker and Kubernetes to deploy and manage the cloud-native components necessary for the benchmark’s workloads.

Learn more at https://en.wikipedia.org/wiki/Cloud_native_computing.

Infrastructure as a service (IaaS)

The term infrastructure as a service (IaaS) refers to a type of cloud service that provides access to virtualized, full-stack computing infrastructure over the internet. IaaS services allow organizations to use dedicated or shared on-demand server/data center infrastructure while avoiding some of the costs of planning, purchasing, and maintaining an equivalent local setup. CloudXPRT allows users to test full IaaS stacks with Amazon Web Services, Google Cloud Platform, and Microsoft Azure.

Learn more at https://en.wikipedia.org/wiki/Infrastructure_as_a_service.

Multi-tier application

A multi-tier application comprises services that are distributed among more than one physical or virtual layer. Most commonly, multi-tier workload architectures have three layers. The CloudXPRT web microservices workload utilizes separate web, application, and data layers to simulate a three-layer, multi-tier application.

Learn more at https://en.wikipedia.org/wiki/Multitier_architecture.

Private cloud

A private cloud is a physically or logically isolated cloud infrastructure system that can be hosted internally (on premises) or externally (off premises). Private cloud users access the private cloud’s applications and services through a private network.

Learn more at https://en.wikipedia.org/wiki/Cloud_computing#Private_cloud.

Public cloud

Public clouds are externally hosted cloud infrastructure systems that often share cloud service provider resources with other users. Public cloud users may access cloud applications and services through the public internet or a direct connection service.

Learn more at https://en.wikipedia.org/wiki/Cloud_computing#Public_cloud.

Hybrid cloud

Hybrid clouds are cloud infrastructure systems that comprise a mixture of on-premises, private cloud, and/or public cloud components. Businesses can use hybrid cloud solutions to optimize the balance of factors such as scalability, flexibility, on-demand performance, security, and cost.

Learn more at https://en.wikipedia.org/wiki/Cloud_computing#Hybrid_cloud.

Microservices

There is no set definition for what constitutes a microservice, but in the context of cloud computing, the term microservices generally refers to relatively lightweight, highly specialized, and loosely connected service components. Microservices are often particularly well suited for virtual systems due to their modularity and scalability.

Learn more at https://en.wikipedia.org/wiki/Microservices.

Web microservices

In the web microservices workload, a simulated user logs into a web application that does three things: provides a selection of stock options, performs Monte-Carlo simulations with those stocks, and presents the user with options that may be of interest. This scenario enables the workload to model a traditional three-tier web application with services in the web, application, and data layers. The workload uses Kubernetes, Docker, NGNIX, REDIS, Cassandra, and monitoring modules to create an end-to-end IaaS scenario. The workload reports performance in transactions per second, which reflects the number of successful requests per second the stack achieves for each level of concurrency. Testers can use this workload’s metrics to compare IaaS stack performance and to evaluate whether any given stack is capable of meeting SLA thresholds.

Data analytics

The data analytics workload calculates XGBoost model training time. XGBoost is a gradient-boosting framework that data scientists often use for machine learning (ML)-based regression and classification problems. The purpose of the workload in the context of CloudXPRT is to measure how well an IaaS stack enables XGBoost to speed and optimize model training. To do this, the data analytics workload uses Kubernetes, Docker, object storage, message pipeline, and monitoring modules to create an end-to-end IaaS scenario. The workload reports latency in the form of job duration in seconds, and throughput rate in the form of jobs per minute. Testers can use this workload’s metrics to compare IaaS stack performance and to evaluate whether any given stack is capable of meeting SLA thresholds.

The CloudXPRT workloads are designed to stress the target infrastructure’s CPU and memory resources. As each workload scales, CPU speed, the number of available CPU sockets and threads, and the amount of available memory and speed of memory access all influence test results.