Compatibility, Quality and Suitability: A measure of performance

You might have heard this from me before. When "getting applications to work" you need to consider 3 elements;

1. Application Compatibility
2. Platform Suitability
3. Performance Quality

We quite often talk to the topics of compatibility and suitability (mostly about virtualisation) in this blog, but there is another dimension that we need to consider; Quality. Or, in this case application performance.

I was speaking with the nice chaps from the Transaction Processing Performance Council (TPC) last week about measuring application performance (how do you do it across platforms and different hardware from different vendors. Essentially, the TPC is non-profit organisation that draws its ideas, code-base and personnel from it's members - of course DELL is a significant and contributing member of the TPC.

A quick overview of the TPC includes:

• The TPC is the only organization that provides consistent price-performance scores
• All tests require full documentation of the components and applications under test, so that the test can be replicated
• The TPC requires an independent audit of results prior to publication
• The TPC tests the whole system performance, not just one piece
• The TPC is database-agnostic: Oracle, IBM DB2, Sybase, Microsoft SQL Server, MySQL, ParAccel and others
• The TPC provides cross-platform performance comparisons, a view of processor vs. real performance, technology comparisons and actual cost of performance comparisons

And some of the TPC key concepts include;

• Run the same 3 copies of a TPC Benchmark
• 3 TPC-C, 3 TPC-E, 3 TPC-H or 3 TPC-DS
• Each contributes equally to the VMS workload
• VMS Metric is min of the 3 benchmark metrics
• Min should equal average if all contribute equally
• Easy to specify, no detailed wording needed to require all 3 TPC Benchmarks contribute equally

Currently, the TPC offers the following benchmarks;

• TPC-C: Online transaction processing (distribution centers)
• TPC-E: Online transaction processing (brokerage)
• TPC-H: Decision support for ad hoc queries
• TPC-DS: Decision-Support benchmark (retailer)

And, just recently the TPC has released a new benchmark for measuring the performance of virtual systems. Here is a quick quote from their press release, 

"The Transaction Processing Performance Council (TPC) announced the availability of TPC-VMS, a specification which enhances the TPC’s existing suite of benchmarks." 

The new Virtual TPC-VMS specification was designed with the following goals:

• Provide a consolidated system workload for three distinct databases running concurrently in a virtualized environment
• Define virtualization metrics that are based on existing TPC benchmark standards
• Outline requirements for measurement and disclosure
• Enable the industry to leverage existing TPC benchmark standards without requiring extensive implementation changes

This is all great stuff, as we move our processing tasks onto virtual platforms, we need to ensure that they work, but also work well. I will soon post more about these measurement concepts and how we get applications to work in the cloud - And, most importantly, how we can then measure their performance.

References:

TPC Home Page

http://www.tpc.org/