VMmark® is a free tool used to measure the performance and scalability of virtualization platforms.

VMware VMmark 2.5

Application-Centric Benchmarking of Real-World Workloads

VMmark uses workloads representative of the applications commonly found in the data center, such as email servers, databases, and so on. VMware has worked closely with its partners to design and implement the benchmark across various software and hardware platforms and has gathered extensive customer feedback to understand how these applications are typically used in virtualized environments. In this way VMmark measures performance using well-understood, existing workloads with which customers are already familiar.

Unique Tile-Based Implementation

The unit of work for a virtualized data center can be usefully defined as a collection of virtual machines executing a set of diverse workloads. The VMmark benchmark refers to this unit of work as a "tile." The virtual machines that make up a VMmark tile, driven by a client system associated with that tile, perform a variety of tasks, both internally and by interacting with the client system and other virtual machines in the tile.

In VMmark 2.x, the total number of tiles that multiple systems in the data center can accommodate, while administrative operations are performed in the background, provides a coarse measure of that data center's consolidation capacity. The performance of the workloads within those tiles provides a fine measure of the data center's overall performance and, combined with the performance of the administrative operations, is used to calculate a VMmark benchmark score.

Multi-Server Virtualized Data Center Benchmarking

The rapid pace of innovation has quickly transformed typical server usage by enabling easier virtualization of bursty and heavy workloads, dynamic virtual machine relocation (vMotion), dynamic datastore relocation (storage vMotion), and automation of many provisioning and administrative tasks across large-scale multi-host environments. In this paradigm, a significant proportion of the stresses on the CPU, network, disk and memory subsystems can be generated by the underlying infrastructure operations. Load balancing across multiple hosts can also greatly affect application performance. Any relevant benchmarking methodology must still focus on user-centric application performance while accounting for the effects of this infrastructure activity on overall platform performance. VMmark 2.x generates a realistic measure of platform performance by incorporating a variety of platform-level workloads such as virtual machine migration, clone and deploy, and storage migration operations, in addition to traditional application-level workloads.

High-Precision Scoring Methodology

During a VMmark benchmark run, which lasts at least three hours, individual performance metrics are collected every 60 seconds. Each of these metrics represents the performance of an individual application or infrastructure workload.

The application workload metrics for each tile are computed and aggregated into a score for that tile by normalizing the different performance metrics, such as MB/second or database commits/second, with respect to a reference system. A geometric mean of the normalized scores is then computed as the final score for the tile. Finally, the resulting per-tile scores are summed to create the application workload portion of the final metric.

A similar calculation is used to create the infrastructure workload portion of the final metric except that, unlike the application workloads, the infrastructure workloads are not scaled explicitly by the user. Consequently, the infrastructure workloads are compiled as a single group and no multi-tile sums are required.

The final benchmark score is computed as a weighted average: 80% to the application workload component and 20% to the infrastructure workload component. These weights were chosen to reflect the relative contribution of infrastructure and application workloads to overall resource demands.

In order for the resultant benchmark score to be considered compliant, the benchmark run must also meet a number of conditions, including minimum quality-of-service requirements.

In addition to the overall benchmark score, a VMmark 2.x full disclosure report also includes the raw and normalized results for each underlying workload and complete details of the virtualization platform configuration. In some cases, studying the workload metrics along with the platform configuration can provide insight into system performance and scaling.

Power Measurement

Power and cooling expenses are a substantial — and increasing — part of the cost of running a data center. Additionally, environmental considerations are a growing factor in data center design and selection. To address these issues, VMmark 2.5 adds optional power measurement to the performance measurements provided by previous VMmark versions. VMmark 2.5 benchmark results can be any of three types:

  • Performance only (no power measurement)
  • Performance with server power
  • Performance with server and storage power

VMmark results with power measurement allow purchasers to see not just absolute performance, but also absolute power consumption and performance per kilowatt. This makes it possible to consider both capital expenses and operating expenses in the selection of new datacenter components.

Note: Though VMmark uses the SPEC® PTDaemon, VMmark results are not SPEC metrics and cannot in any manner be compared to SPEC metrics.

Power Measurement