TPC-VMS Benchmark To Compare Virtual Workloads

This past November the Transaction Processing Performance Council (the TPC) released a new benchmark, the TPC-VMS. If you are interested in the details you can download the PDF by clicking here.

The TPC benchmarks have been the industry standard when it comes to benchmarking database performance. As more companies move towards virtualization solutions the standard TPC benchmarks were starting to become irrelevant. Rather than let themselves become obsolete the TPC decided to embrace the changing landscape and thus defined a new test for database performance in a virtual environment. The idea is very simple: you configure three virtual machines running an identical benchmark and then consolidate those three instances onto one host and run the TPC-VMS test.

For example, let’s say we have a host with three guests, each running a version of SQL Server. You would configure each of those guests to run a standard TPC test, let’s use TPC-E. We would run the TPC-E on each guest at the same time. The slowest performing instance would then be reported as our VMStpsE benchmark, one of four new benchmarks (VMStpmC, VMSQphH, and VMSQphDS are the others) which have no relation to each other nor to the base benchmarks (TPC-C, TPC-E, TPC-H, and TPC-DS) we are used to seeing. The slowest performing of the three virtual machines will be the one reported as your benchmark.

If my explanation didn’t make sense (or if you are a manager reading this) here’s a pretty picture for you:

From the TPC website we can find that the TPC-VMS specification was designed with the following goals:

• Provide a consolidated system workload for three distinct databases running concurrently in a virtualized environment
• Define virtualization metrics that are based on existing TPC benchmark standards
• Provide for repeatable measurements
• Provide requirements for disclosure and documentation of the measurements to ensure compliance with this
  specification
• Enable the industry to leverage existing TPC benchmark standards without requiring extensive implementation changes

I also found some mentions in various articles online that Red Hat has plans to provide TPC-VMS results however no results have been published. I cannot find mention of VMWare, Microsoft, or Oracle announcing any plans to publish results either. My guess is that they will likely wait for Red Hat to get theirs done and draw the line in the sand for everyone else to beat. Red Hat has released some benchmarks for the SPECvirt test (a benchmark derived from the VMmark test by VMWare).

I had heard about the TPC-VMS last month but forgotten about it until I got an email yesterday from a my friend who shared his thoughts were about the importance of the TPC-VMs benchmarks:

1. It will likely lead to a direct shoot out between the virtualization giants VMWare, Red Hat, Microsoft (HYPER-V), Oracle, Xen.
2. Since this is the TPC all of the big guns (VMWare, DELL, HP, Oracle, Microsoft, Red Hat) will have their best working on extracting the most out of their platforms and software.
3. Since results are published there are often some interesting performance tips hidden in the documentation.
4. Now we are likely to have a very easy way to compare metal to virtual database implementations.

I believe this new benchmark will be a good thing overall. Now I’m wondering how soon before we get a benchmark for host (or Cloud) service providers such as Amazon, Rackspace, and Microsoft. If it took this long to get a TPC-VMS benchmark agreed upon, how long before we will see TPC-CLOUD?