BenchmarkXPRT Blog banner

Category: CloudXPRT

More information about the CloudXPRT results submission process

Earlier this month, we discussed the possibility of using a periodic results submission process for CloudXPRT instead of the traditional rolling publication process that we’ve used for the other XPRTs. We’ve received some positive responses to the idea, and while we’re still working out some details, we’re ready to share the general framework of the process we’re planning to use.

  • We will establish a results review group, which only official BenchmarkXPRT Development Community members can join.
  • We will update the CloudXPRT database with new results once a month, on a pre-published schedule.
  • Two weeks before each publication date, we will stop accepting submissions for consideration for that review cycle.
  • One week before each publication date, we will send an email to the results review group that includes the details of that month’s submissions for review.
  • The results review group will serve as a sanity check process and a forum for comments on the month’s submissions, but we reserve the right of final approval for publication.
  • We will not restrict publishing results outside of the monthly review cadence, but we will not automatically add those results to the results database.
  • We may add externally published results to our database, but will do so only after vetting, and only on the designated day each month.

Our goal is to strike a balance between allowing the tech press, vendors, or other testers to publish CloudXPRT results on their own schedule, and simultaneously building a curated results database that OEMs or other parties can use to compete for the best results.

We’ll share more details about the review group, submission dates, and publications dates soon. Do you have questions or comments about the new process? Let us know what you think!

Justin

CloudXPRT is up next, and we’re thinking about how to handle results submission and publication

Last month, we provided an update on the CloudXPRT development process and more information about the three workloads that we’re including in the first build. We’d initially hoped to release the build at the end of April, but several technical challenges have caused us to push the timeline out a bit. We believe we’re very close to ready, and look forward to posting a release announcement soon.

In the meantime, we’d like to hear your thoughts about the CloudXPRT results publication process. Traditionally, we’ve published XPRT results on our site on a rolling basis. When we complete our own tests, receive results submissions from other testers, or see results published in the tech media, we authenticate them and add them to our site. This lets testers make their results public on their timetable, as frequently as they want.

Some major benchmark organizations use a different approach, and create a schedule of periodic submission deadlines. After each deadline passes, they review the batch of submissions they’ve received and publish all of them together on a single later date. In some cases, they release results only two or three times per year. This process offers a high level of predictability. However, it can pose significant scheduling obstacles for other testers, such as tech journalists who want to publish their results in an upcoming device review and need official results to back up their claims.

We’d like to hear what you think about the different approaches to results submission and publication that you’ve encountered. Are there aspects of the XPRT approach that you like? Are there things we should change? Should we consider periodic results submission deadlines and publication dates for CloudXPRT? Let us know what you think!

Justin

Make confident choices about your company’s future tech with the XPRTs

Durham, NC, April 23, 2020 — Principled Technologies and the BenchmarkXPRT Development Community have released a video on the benefits of consulting the XPRTs before committing to new technology purchases.

AIXPRT, one of the battery of XPRT benchmark tools, runs image-classification and object-detection workloads to determine how well tech handles AI and machine learning.

CloudXPRT, another XPRT tool, accurately measures the end-to-end performance of modern, cloud-first applications deployed on infrastructure as a service (IaaS) platforms – allowing corporate decision-makers to select the best configuration for every objective.

All of the XPRTs give companies the real-world information necessary to determine which prospective future tech p – and which will disappoint

According to the video, “The XPRTs don’t just look at specs and features; they gauge a technology solution’s real-world performance and capabilities. So you know whether switching environments is worth the investment. How well solutions support machine learning and other AI capabilities. If next-gen releases beat their rivals or fall behind the curve.”

Watch the video at facts.pt/pyt88k5. To learn more about how AIXPRT, CloudXPRT, WebXPRT, MobileXPRT, TouchXPRT, CrXPRT, and HDXPRT can help IT decision-makers can make confident choices about future purchases, go to www.BenchmarkXPRT.com.

About Principled Technologies, Inc.
Principled Technologies, Inc. is the leading provider of technology marketing and learning & development services. It administers the BenchmarkXPRT Development Community.

Principled Technologies, Inc. is located in Durham, North Carolina, USA. For more information, please visit www.principledtechnologies.com.

Company Contact
Justin Greene
BenchmarkXPRT Development Community
Principled Technologies, Inc.
1007 Slater Road, Suite #300
Durham, NC 27703
BenchmarkXPRTsupport@PrincipledTechnologies.com

Adapting to a changing tech landscape

The BenchmarkXPRT Development Community started almost 10 years ago with the development of the High Definition Experience & Performance Ratings Test, also known as HDXPRT. Back then, we distributed the benchmark to interested parties by mailing out physical DVDs. We’ve come a long way since then, as testers now freely and easily access six XPRT benchmarks from our site and major app stores.

Developers, hardware manufacturers, and tech journalists—the core group of XPRT testers—work within a constantly changing tech landscape. Because of our commitment to providing those testers with what they need, the XPRTs grew as we developed additional benchmarks to expand the reach of our tools from PCs to servers and all types of notebooks, Chromebooks, and mobile devices.

As today’s tech landscape continues to evolve at a rapid pace, our desire to play an active role in emerging markets continues to drive us to expand our testing capabilities into areas like machine learning (AIXPRT) and cloud-first applications (CloudXPRT). While these new technologies carry the potential to increase efficiency, improve quality, and boost the bottom line for companies around the world, it’s often difficult to decide where and how to invest in new hardware or services. The ever-present need for relevant and reliable data is the reason many organizations use the XPRTs to help make confident choices about their company’s future tech.

We just released a new video that helps to explain what the XPRTs provide and how they can play an important role in a company’s tech purchasing decisions. We hope you’ll check it out!

We’re excited about the continued growth of the XPRTs, and we’re eager to meet the challenges of adapting to the changing tech landscape. If you have any questions about the XPRTs or suggestions for future benchmarks, please let us know!

Justin

News about the CloudXPRT source code

For much of the BenchmarkXPRT Development Community’s history, we offered community members exclusive access to XPRT benchmark source code. Back in February, we started to experiment with a different approach when we made the AIXPRT source code publicly available on GitHub. By allowing anyone who is interested in AIXPRT to download and review the source code, we reinforced our commitment to making the XPRT development process as transparent as possible. We also want the XPRTs to continue to contribute to fair practices in the benchmarking world, and we believe that expanded access to the source code encourages constructive feedback to help in this goal.

The feedback we received after publishing the AIXPRT source code was very positive; thank you to all who reached out. Because of that feedback and our desire to increase openness, we’ve decided use standard open source licenses to make the CloudXPRT source code available to the public when we release of the first build, or shortly thereafter. As with AIXPRT, folks will be able to download the CloudXPRT source code and submit potential workloads for future consideration, but we reserve the right to control derivative works.

We’ll share more information about the first CloudXPRT release and its source code in the coming weeks. If you have any questions about XPRT source code, feel free to ask.  We also welcome any thoughts about using this approach to release the source code of other XPRT benchmarks. As always, feel free to comment below or reach out by email.

Justin

More details about CloudXPRT’s workloads

About a month ago, we posted an update on the CloudXPRT development process. Today, we want to provide more details about the three workloads we plan to offer in the initial preview build:

  • In the web-tier microservices workload, a simulated user logs in to a web application that does three things: provides a selection of stock options, performs Monte-Carlo simulations with those stocks, and presents the user with options that may be of interest. The workload reports performance in transactions per second, which testers can use to directly compare IaaS stacks and to evaluate whether any given stack is capable of meeting service-level agreement (SLA) thresholds.
  • The machine learning (ML) training workload calculates XGBoost model training time. XGBoost is a gradient-boosting framework  that data scientists often use for ML-based regression and classification problems. The purpose of the workload in the context of CloudXPRT is to evaluate how well an IaaS stack enables XGBoost to speed and optimize model training. The workload reports latency and throughput rates. As with the web-tier microservices workload, testers can use this workload’s metrics to compare IaaS stack performance and to evaluate whether any given stack is capable of meeting SLA thresholds.
  • The AI-themed container scaling workload starts up a container and uses a version of the AIXPRT harness to launch Wide and Deep recommender system inference tasks in the container. Each container represents a fixed amount of work, and as the number of Wide and Deep jobs increases, CloudXPRT launches more containers in parallel to handle the load. The workload reports both the startup time for the containers and the Wide and Deep throughput results. Testers can use this workload to compare container startup time between IaaS stacks; optimize the balance between resource allocation, capacity, and throughput on a given stack; and confirm whether a given stack is suitable for specific SLAs.

We’re continuing to move forward with CloudXPRT development and testing and hope to add more workloads in subsequent builds. Like most organizations, we’ve adjusted our work patterns to adapt to the COVID-19 situation. While this has slowed our progress a bit, we still hope to release the CloudXPRT preview build in April. If anything changes, we’ll let folks know as soon as possible here in the blog.

If you have any thoughts or comments about CloudXPRT workloads, please feel free to contact us.

Justin

Check out the other XPRTs:

Forgot your password?