Cockroach Labs recently issued its 2022 Cloud Report to help customers identify the best infrastructure to run their cloud-native CockroachDB. The report’s insights — the relative strengths of the top 3 cloud providers, the best processors for their cloud instances, pricing, and more — also apply to cloud customers more broadly.
The full report is well worth the read for high-level strategy insights, which we’ll review here, as well as deep-dive details into benchmarks, configurations, and recommendations.
The report aims to give customers insights into the best options among Microsoft Azure, Amazon AWS, and Google Cloud. The report does not name an overall winner or best choice.
The report provides this overview of the methodology:
“Using the product we know best – CockroachDB – we ran an industry-standard-based database benchmark configured consistently across runs and instance types, as well as leading micro-benchmarks in the areas of compute, networking, and storage, to compare the performance and value of 56 instance types and 107 discrete configurations across the three clouds.”
In an interview, the report’s lead author gave additional detail: “We run these because our customers need to know what kinds of infrastructure they need to provision to run a single CockroachDB instance across multiple cloud providers,” says Keith McClellan, Director of Partner Solutions Engineering at Cockroach Labs. “We think this information is valuable for the tech ecosystem.”
Cloud Service Pricing
When it comes to the all-important issue of price, the report found that all three providers have price-competitive offerings and, while there were differences based on configuration, those differences are not major and no one provider had an overall price advantage in this iteration of its test.
“In the past, there were price disparities where there were strong advantages to use one or another on a performance or price basis. We found this year that largely those differences have leveled out.”
Keith McClellan, Director of Partner Solutions Engineering, Cockroach Labs
Customers can get roughly the same price and performance from all three major providers — cost is no longer a major differentiator.
The report notes: “Overall, pricing is very linear, with pricing scaling by the number of vCPUs, GB of RAM, or the amount of disk allocated. In most cases, you can expect to pay roughly the same amount for these resources across all three clouds, at least for all the configurations we considered.”
Cloud Storage Cost Impact
The Cockroach Labs report emphasizes how much of a customer’s total overall cost (“outsized”) is driven by storage, especially when that customer chooses to use high-performance storage, which isn’t necessarily recommended. Close to 70% of total cost across all three providers is high-performance storage when a customer chooses that option.
Even in the event that a customer is using relatively small amounts of persistent block storage, the cost of a workload is influenced more by the cost of the storage than the cost of the instance, the report notes. For this reason, the authors urge customers with persistent workloads to optimize cost calculations based on this factor. This means mid-tier storage solutions are preferable unless the workload requires an extremely high number of input/output operations per second or IOPS.
Cockroach Labs sees a lot of customers who buy the most expensive storage but don’t necessarily need that option. Customers tend to overprovision to ensure they have the capacity and performance they need. “If, for your workload, you need an amount of IOPS and throughput that can be provided by lower-tier storage, you will get the same performance, at least today,” McClellan says. “All providers are now providing great mid-tier options with wonderful performance until you get to extremely high storage bandwidth requirements.”
Processors and Performance
Cockroach Labs’ 2022 report features comparisons of Intel Ice Lake and AMD Milan processors. Instances using AMD Milan took the top overall position for performance in large and small nodes. AMD also outperformed Intel across both OLTP and CPU benchmarks. In previous years, Intel outperformed AMD as the overall performance leader and as the price-for-performance leader. “Customers are getting a great value when they used AMD-based instances,” McClellan says.
Caps, Guarantees, and Throttling
The report offers a measure of caution when it comes to performance caps vs. performance guarantees — noting the need to understand the difference and recognize very specifically what customers are getting from their cloud provider. “In general, when we paid explicitly for a resource we got the promised performance out of that resource.”
For example, the report noted that cross-region throughput was consistent for all three providers, as was storage performance. In cases where the cloud provider advertised a performance range (up to 10 Gbps, for example) and did not explicitly charge for a specific level of performance, the results varied. In certain data transfer situations with a specified range, the range was wide in practice and that included apparent throttling.
“Naively, I might think it’s a perfectly acceptable instance type for my workload. But it turns out I really need higher because I’m not guaranteed more than 30-40% of that total throughput. You could get throttled back which could cause problems you’re not expecting,” McClellan says, adding CockroachDB’s testing showed throttling by all three providers.
Want to gain more insights from Cloud Wars Expo?
Starting on July 20th, more than 40 hours of on-demand cloud education content will be available for free to Acceleration Economy Subscribers.
