Earlier this month Microsoft and GigaOM announced a new benchmark study comparing AWS RDS to Azure SQL Database. This study was authored by the same people that wrote the previous GigaOM data warehouse benchmark last year. I enjoyed the data warehouse study. I found it to be fair and thorough enough to help the reader understand how to conduct their own benchmark testing. I was eager to read the new SQL Transactional Processing Price-Performance Testing study.
I found this latest effort to be a good start, but it fell short of the effort the authors put forth in their previous benchmark for data warehousing.
Before I go any further, I want to thank the authors for putting together their results. I recognize that these are humans, working hard, putting forth their best efforts at being fair and thorough. Comparing cloud services is not an easy task. I found this latest effort to be good, but not great. If they were students of mine I would grade this latest paper from them a solid B-.
Let’s break it down.
The Good Stuff
First, the good stuff. I love how they drove everything towards a formula, price/performance, where performance is tracked in transactions per second. The downside to price/performance is that not every workload is focused on transactions per second. Still, I’d like to see this formula adopted as a standard way of comparing services.
In the past I’ve focused only on total price as shown by the online pricing calculators. This is because (1) you aren’t supposed to publish benchmarks without permission from the company (Microsoft, AWS) and (2) I can’t bankroll this level of test AND maintain my scotch and bacon addictions. By using price/performance you level the playing field somewhat. A service may cost more, but if it runs your query in half the time, the cost may be worth it.
I also liked the choice of using TPC-E as their test, I believe that to be a fair way to compare how the services will handle a workload. And I liked how they explained the difficulties in comparing services and the associated hardware. That’s something I’ve written about previously. Many times, really.
It is frustrating to compare the data services being offered between Azure and AWS. Part of me thinks this is done on purpose by both companies in an effort to win our favor without giving away more information than is necessary. This is a common practice, and I’m not bashing either company for doing what has been done for centuries. I’m here to help others figure out how to make the right choice for their needs. At the end of the day, I believe both Amazon and Microsoft want the same thing: happy customers.
But it is not in their best interest to make it easy for anyone to compare costs. This is how utilities operate. Make no mistake, AWS and Azure are the new electric company.
Now, for the items that I didn’t like as much. I’ll capture the quote from the article and explain my concern.
The Not As Good Stuff
“There are no exact matches in processors or memory.” – This is a bit of nitpicking, but I took issue here with the use of the word “or”. As someone who charges (and receives) top dollar for performing technical reviews of books, it bugged me. The authors are correct in saying that it is hard to find exact matches. However, I can certainly find a match for vCPU, but not for memory. Azure publishes memory as weird increments, starting at 10.2 GB while AWS shows traditional increments of 8, 16, etc. So, yeah, it’s a nitpick. But it was this exact item what caught my eye and made me dig deeper to fact check everything. Warrants mentioning.
“Thus, R4 seemed a suitable instance class to use for AWS RDS.” – The authors explain why they chose R4 (memory optimized instance) versus the M4 (general purpose). I have no issue with this except that neither M5 or R5 was considered. This study just came out, why were those instances not considered? And since the authors went out of their way to tell us what AWS says about the R4, let me tell you what AWS also says about the R5:
“R5 instances deliver 5% additional memory per vCPU than R4 and the largest size provides 768 GiB of memory. In addition, R5 instances deliver a 10% price per GiB improvement and a ~20% increased CPU performance over R4.”
I can’t think of any reason why the authors chose R4 here. But let’s move past this, because now is time for the hard part: finding a suitable match for Azure SQL Database.
“On the Azure side, we expect customers to gravitate towards SQL Database Business Critical (BC) offerings…” – Well, Azure doesn’t offer a memory optimized version of SQL Database, so I guess using BC is fine. But the question I have now is why not consider using Managed Instance? In the data warehouse benchmark study they tried a variety of sizes against the workload. This study focused ONLY on one size machine. This is part of the reason they got a B-, they weren’t thorough enough for my liking. I’d send them back and tell them to run more tests against different sized machines and include Managed Instance. At the very least they could have made an effort to simply use general purpose, it would have been closer to an apples-to-apples comparison.
“Therefore, we chose the BC_Gen5_80 instance, which has more CPUs than R4.16xlarge, but less memory at 408 GB.” – Yes, finding an exact match is difficult. Here’s a breakdown of what they chose:
But this image shows AWS at 64,000 provisioned IOPS, and further in the study they say they tested against 32,000 provisioned IOPS. So, which is it? I’ve no idea. Neither do you. But I do know that provisioning 32,000 IOPS added about $6k to the monthly bill.
“…the monthly cost of Microsoft Azure comes to $40,043.71. The monthly cost for AWS comes to $65,239.43.” – Verified, I can get the same prices using the AWS and Azure calculators. But the small detail that is glossed over here is single versus multi-zone. The AWS calculator is clear, if you deploy multi-zone, the price doubles. The Azure calculator doesn’t have this option, it only exists when you create your SQL Database. I’d be shocked to find out that deploying multi-zone in Azure didn’t bump the price as well. But the chart above clearly states “in a single availability zone”. So, which is it?
I’ve no idea. Neither do you.
Summary
Some quick math tells me that if we drop the multi-zone from AWS RDS the price/performance result comes in at $1,269.85, slightly cheaper than the $1,410.04 for SQL Database. And this is why I like price/performance as a metric. A database service may have a slightly higher price, but offers greater throughput.
This was the exact conclusion from the data warehouse study, too. The cost for Azure SQL Data Warehouse was just a tad more than AWS Redshift, but the performance with Azure was better. I wanted to see a similar conclusion in this study.
Instead, we have a report with a handful of inaccuracies. Perhaps in an effort to rush to publish ahead of Ignite, they simply used a wrong graph, or missed doing one final round of edits. When you are doing this work it is easy to have such things fall through the cracks.
I’d love to see this study republished without these errors. I’d also love for AWS and Azure to find a way to make it easier to compare costs and services.
REFERENCES:
Azure vs. AWS Data Services Comparison
Updated Data Services Comparison: AWS vs. Azure
Azure vs AWS Analytics and Big Data Services Comparison
Updated Analytics and Big Data Comparison: AWS vs. Azure
Azure SQL Data Warehouse Costs vs AWS Redshift
Azure pricing calculator
AWS pricing calculator
Amazon EC2 Instance Types
Sizes for Windows virtual machines in Azure
Azure SQL Database pricing
Data Warehouse in the Cloud Benchmark
SQL Transactional Processing Price-Performance Testing
Tom,
Nice post, and I agree with your arguments, and would like to add another one. I don’t find that customers choose cloud platforms based on a single metric like price per performance. It’s very much a holistic discussion, and in that sense, Microsoft offers a lot of really beneficial enterprise features (like ease of integration with AD/AAD), that are much more appealing then a nebulous price/perf comparison.
Thanks Joey, and I agree that there is more to making a choice than just one metric.
What is your opinion on the fact that the DW benchmark study was sponsored by Microsoft. Wouldn’t that make the study a bit biased?
Just playing devil’s advocate. I know you said that the study was “fair and thorough”.
Both studies were sponsored by Microsoft. I don’t have any reason to not trust GigaOM here. I did find the DW study to be fair and more thorough than the SQL Database study. Even if the end result has some bias, the authors were clear in both paper that the reader should conduct their own independent testing before making a decision.
Thanks for your insights Tom.
I found this tidbit concerning: An Issue with caching:
“In addition to scaling process and storage resources separately, Azure Synapse Analytics stands out for its result caching capability (it has a fully managed 1 TB cache). Thus, when a query is made it is stored in this cache to speed up the next query that consumes the same type of data.”
https://blog.bismart.com/en/azure-synapse-difference-from-azure-data-bricks-and-sql
From the study:
” Three power runs were completed. Each of the 22 queries was executed three times in order (1, 2, 3,…) against each vendor cloud platform, and the fastest of the three times was used as the performance metric. ”
This essentially means that the performance of cache was measured and the metric derived is not price/DW performance but price/cache performance. This would approach accurate in scenarios where percent repeat queries approaches 100% of total query volume. But I doubt this was the intend, otherwise why talk about vcpu and memory parameters at all? I am also not clear about why some query results don’t seem to be cached effectively and make an appearance of something being processed (duration is just too long for cache access only operation).
I hope someone can shed some light on this topic and maybe offer an alternative study that benchmarks both cached and non-cached performance.