VMmark 1.x - VMware Products
The Need for a Standard Virtualization Benchmark
Conventional server benchmarks in use today were designed to measure the performance of a single application running on a single operating system, inside a single physical computer. However, these benchmarks cannot accurately measure the performance of a single application running inside a virtual machine, nor can they measure the performance of a single hardware platform used to run several virtual machines at once. As a result, there is increasing interest in a more effective solution for measuring the performance and scalability of virtualized IT environments.
VMmark: The Industry's First Virtualization Benchmark
VMmark is a free tool that hardware vendors, virtualization software vendors and other organizations use to measure the performance and scalability of applications running in virtualized environments. This virtualization benchmark software features a novel tile-based scheme for measuring application performance and provides a consistent methodology that captures both the overall scalability and individual application performance
Traditional Benchmarking vs. VMmark Tile-Based Benchmarking

VMmark: The First Industry-Standard Virtualization Benchmark
VMware developed VMmark as a standard methodology for comparing virtualized systems. The benchmark system in VMmark comprises a series of “sub-tests” that are derived from commonly used load-generation tools, as well as from benchmarks developed by the Standard Performance Evaluation Corporation (SPEC®). In parallel to VMmark, VMware is a member of the SPEC Virtualization subcommittee and is working with other SPEC members to create the next-generation virtualization benchmark.
FEATURES
Open Standards and Platform Neutrality
VMmark is not a commercial or proprietary product, but rather a product of our commitment to the development of open standards for virtualization benchmarks. As such, the VMmark software is agnostic towards individual hardware platforms and virtualization software systems so that users can get an objective measurement of virtualization performance.
VMware is working with the Standard Performance Evaluation Corporation (SPEC®) - a non profit organization that establishes, maintains and endorses standardized benchmarks - and members of the SPEC Virtualization subcommittee to develop standard methods of comparing virtualization performance for data center servers.
Other current participants include AMD, Dell, Fujitsu Siemens, Hewlett-Packard, Intel, IBM, Microsoft, Red Hat, Sun Microsystems and SWsoft.
Application-Centric Benchmarking of Real-World Workloads
VMmark uses workloads representative of those applications most often found in the data center, such as email servers, databases, etc. VMware has worked closely with its partners to design and implement the benchmark across various software and hardware platforms, and has also gathered extensive customer feedback to understand how these applications are typically used in virtualized environments.
To measure performance, VMmark leverages well-understood, existing benchmarks that customers are already familiar with.
VMmark incorporates 64-bit versions of three of the VMmark workloads—Java Server, Database Server and Web Server—to reflect the growing use of 64-bit applications and operating systems.
Unique Tile-Based Implementation
The unit of work for a benchmark of virtualized consolidation environments can be naturally defined as a collection of virtual machines executing a set of diverse workloads. The VMmark benchmark refers to this unit of work as a tile.
The total number of tiles that a system can accommodate provides a coarse-grain measure of that system's consolidation capacity. This concept is similar to some server benchmarks, such as TPC-C, that scale the workload in a step-wise fashion to increase the system load.
High-Precision Scoring Methodology
VMmark allows for the integration of the different component metrics into an overall score. Once a VMmark test completes, each individual workload reports its relevant performance metric. These metrics are collected at frequent intervals during the course of a run. A VMmark benchmark test is designed to run for at least three hours with workload metrics reported every 60 seconds.
After a benchmark run, the workload metrics for each tile are computed and aggregated into a score for that tile. This aggregation is performed by first normalizing the different performance metrics such as MB/second and database commits/second with respect to a reference system. Then, a geometric mean of the normalized scores is computed as the final score for the tile. The resulting per-tile scores are then summed to create the final metric.
This approach helps users measure the virtualization overhead of the individual application workloads, as well as the scalability of the entire system.
VMMARK SYSTEM REQUIREMENTS
Workload Software
Operating systems: Microsoft Windows Server 2003 Release 2 Enterprise Edition (32-bit and 64-bit); SUSE Linux Enterprise Server 10 (32-bit and 64-bit)
Applications: Microsoft Exchange 2003; SPECjbb2005; SPECweb2005; MySQL; dbench
Client Software
Each VMmark tile requires a client machine. These client machines run Microsoft Windows Server 2003 Release 2 Enterprise Edition (32-bit) with the following applications:VMmark harness
- VMmark harness
- STAF framework and STAX execution engine
- LoadSim 2003
- Microsoft Outlook 2003 (standalone or included in Microsoft Office 2003)
- BEA JRockit 5.0 JVM JDK
- SPECweb client (included with the SPECweb2005 package)
- MySQL database server
- SysBench database benchmark
- SPECjbb monitor (included with the SPECjbb2005 package)
Free Software
You can download the following free software packages from the web:
- SUSE Linux Enterprise Server 10
- Apache Ant build tool
- Apache HTTP Server
- APC cache for PHP
- BEA JRockit 5.0 JVM JDK
- Cygwin environment
- MySQL database server
- SysBench database benchmark
- Dbench (a customized version included with VMmark software)
- FastCGI CGI extension
- MySQL database
- SysBench database benchmarking tool
- PHP scripting language
- STAF/STAX software
- VMware VMmark Harness
- VMware memory soaker
Evaluation Software
You can download evaluation versions of the following software packages from the web:
- Microsoft Outlook 2003 (standalone or included in Microsoft Office 2003)
RESULTS
Get an accurate measurement of application performance in virtualized environments with VMware VMmark, the industry's first virtualization benchmark for x86-based computers. Measure virtual machine performance accurately and reliably.
64 Cores
TestSubmitter | System Description | VMmark Version & Score | Processors | Published Date |
---|---|---|---|---|
NEC | NEC Express5800/A1160 VMware ESX v4.0 |
VMmark v1.1.1 48.23 @ 32 tiles View Disclosure |
16 sockets 64 total cores 64 total threads |
11/17/09 |
48 Cores
Submitter | System Description | VMmark Version & Score | Processors | Published Date |
---|---|---|---|---|
Dell | Dell PowerEdge R815 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 59.74@41 tiles View Disclosure |
4 sockets 48 total cores 48 total threads |
08/10/10 |
HP | HP ProLiant DL785 G6 VMware ESX v4.0 |
VMmark v1.1.1 53.73 @ 35 tiles View Disclosure |
8 sockets 48 total cores 48 total threads |
08/25/09 |
HP | HP ProLiant DL785 G6 VMware ESX v4.0 |
VMmark v1.1.1 47.77 @ 30 tiles View Disclosure |
8 sockets 48 total cores 48 total threads |
08/11/09 |
NEC | NEC Express5800/A1160 VMware ESX v4.0 |
VMmark v1.1.1 34.05@24 tiles View Disclosure |
8 sockets 48 total cores 48 total threads |
07/28/09 |
IBM | IBM System x3950 M2 VMware ESX v4.0 |
VMmark v1.1 33.85@24 tiles View Disclosure |
8 sockets 48 total cores 48 total threads |
06/16/09 |
32 Cores
Submitter | System Description | VMmark Version & Score | Processors | Published Date |
---|---|---|---|---|
Fujitsu | Fujitsu RX600 S5 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 77.29@51 tiles View Disclosure |
4 sockets 32 total cores 64 total threads |
10/19/10 |
Cisco | Cisco UCS C460 M1 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 76.10@51 tiles View Disclosure |
4 sockets 32 total cores 64 total threads |
09/07/10 |
Fujitsu | Fujitsu RX600 S5 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 75.77@50 tiles View Disclosure |
4 sockets 32 total cores 64 total threads |
06/29/10 |
HP | HP ProLiant DL580 G7 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 75.01@50 tiles View Disclosure |
4 sockets 32 total cores 64 total threads |
08/24/10 |
Dell | Dell PowerEdge R910 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 74.34@50 tiles View Disclosure |
4 sockets 32 total cores 64 total threads |
07/27/10 |
Cisco | Cisco UCS C460 M1 VMware ESXi v4.0 Update 2 |
VMmark v1.1.1 73.82@50 tiles View Disclosure |
4 sockets 32 total cores 64 total threads |
05/04/10 |
Fujitsu | Fujitsu RX600 S5 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 73.69@50 tiles View Disclosure |
4 sockets 32 total cores 64 total threads |
06/15/10 |
Lenovo | Lenovo WQ R680 G7 VMware ESXi v4.0 Update 2 |
VMmark v1.1.1 73.20@50 tiles View Disclosure |
4 sockets 32 total cores 64 total threads |
07/28/10 |
IBM | IBM System x3850 X5 VMware ESXi v4.0 Update 2 |
VMmark v1.1.1 71.85@49 tiles View Disclosure |
4 sockets 32 total cores 64 total threads |
04/20/10 |
Cisco | Cisco UCS B440 M1 VMware ESXi v4.1 |
VMmark v1.1.1 71.13@48 tiles View Disclosure |
4 sockets 32 total cores 64 total threads |
07/27/10 |
IBM | IBM System x3850 X5 VMware ESXi v4.0 Update 2 |
VMmark v1.1.1 70.78@48 tiles View Disclosure |
4 sockets 32 total cores 64 total threads |
04/06/10 |
Dell | Dell PowerEdge R810 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 59.21@41 tiles View Disclosure |
4 sockets 32 total cores 64 total threads |
08/10/10 |
Dell | Dell PowerEdge M910 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 58.37@41 tiles View Disclosure |
4 sockets 32 total cores 64 total threads |
08/24/10 |
HP | HP ProLiant DL785 G5 VMware ESX v4.0 |
VMmark v1.1 31.56@21 tiles View Disclosure |
8 sockets 32 total cores 32 total threads |
06/02/09 |
Unisys | Unisys ES7000 Model 7405R VMware ESX v4.0 |
VMmark v1.1 30.86@20 tiles View Disclosure |
8 sockets 32 total cores 32 total threads |
05/19/09 |
HP | HP ProLiant DL785 G5 VMware ESX v4.0 |
VMmark v1.1 30.50@21 tiles View Disclosure |
8 sockets 32 total cores 32 total threads |
04/21/09 |
Unisys | Unisys ES7000 Model 7405R VMware ESX v3.5.0 Update 3 |
VMmark v1.1 29.19@19 tiles View Disclosure |
8 sockets 32 total cores 32 total threads |
04/07/09 |
Sun | Sun Fire X4600 M2 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 29.11@19 tiles View Disclosure |
8 sockets 32 total cores 32 total threads |
03/10/09 |
Unisys | Unisys ES7000 Model 7405R VMware ESX v3.5.0 Update 3 |
VMmark v1.1 28.97@19 tiles View Disclosure |
8 sockets 32 total cores 32 total threads |
03/05/09 |
HP | HP ProLiant DL785 G5 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 27.71@19 tiles View Disclosure |
8 sockets 32 total cores 32 total threads |
12/19/08 |
IBM | IBM System x3950 M2 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 24.62@18 tiles View Disclosure |
8 sockets 32 total cores 32 total threads |
10/02/08 |
Unisys | Unisys ES7000 Model 7405R G1 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 21.96@15 tiles View Disclosure |
8 sockets 32 total cores 32 total threads |
11/05/08 |
HP | HP ProLiant DL785 G5 VMware ESX v3.5.0 Update 1 |
VMmark v1.1 21.88@16 tiles View Disclosure |
8 sockets 32 total cores 32 total threads |
08/18/08 |
24 Cores
Submitter | System Description | VMmark Version & Score | Processors | Published Date |
---|---|---|---|---|
Fujitsu | Fujitsu RX600 S5 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 55.88@ 39 tiles View Disclosure |
4 sockets 24 total cores 48 total threads |
10/19/10 |
Fujitsu | Fujitsu BX960 S1 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 52.27@35 tiles View Disclosure |
4 sockets 24 total cores 48 total threads |
06/29/10 |
Dell | Dell PowerEdge R715 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 32.44@22 tiles View Disclosure |
2 sockets 24 total cores 24 total threads |
09/21/10 |
HP | HP ProLiant DL385 G7 VMware ESX v4.0 Update 1 |
VMmark v1.1.1 30.96@22 tiles View Disclosure |
2 sockets 24 total cores 24 total threads |
05/04/10 |
HP | HP ProLiant DL385 G7 VMware ESX v4.0 Update 1 |
VMmark v1.1.1 30.60@21 tiles View Disclosure |
2 sockets 24 total cores 24 total threads |
04/20/10 |
HP | HP ProLiant DL585 G6 VMware ESX v4.0 |
VMmark v1.1.1 29.95@20 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
07/14/09 |
Dell | Dell PowerEdge R905 VMware ESX v4.0 |
VMmark v1.1.1 29.51@20 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
07/28/09 |
HP | HP ProLiant BL685c G6 VMware ESX v4.0 |
VMmark v1.1.1 29.19@20 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
07/14/09 |
IBM | IBM System x3850 M2 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 20.50@14 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
03/24/09 |
IBM | IBM System x3850 M2 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 20.41@14 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
02/24/09 |
Dell | Dell PowerEdge R900 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 19.99@14 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
02/10/09 |
Inspur | Inspur NF520D2 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 19.67@14 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
01/27/09 |
Sun | Sun Fire X4450 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 19.47@14 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
01/13/09 |
IBM | IBM System x3850 M2 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 19.10@14 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
09/17/08 |
Dell | Dell PowerEdge R900 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 18.69@14 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
12/02/08 |
HP | HP ProLiant BL680c G5 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 18.64@14 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
03/30/09 |
HP | HP ProLiant DL580 G5 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 18.56@14 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
10/06/08 |
Dell | Dell PowerEdge R900 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 18.49@14 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
09/12/08 |
HP | HP ProLiant BL680c G5 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 16.05 @ 12 tiles View Disclosure |
4 sockets 24 total cores 24 total threads |
09/23/08 |
16 Cores
Submitter |
System Description | VMmark Version & Score | Processors | Published Date |
---|---|---|---|---|
Fujitsu | Fujitsu RX600 S5 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 40.52@28 tiles View Disclosure |
2 sockets 16 total cores 32 total threads |
10/19/10 |
Fujitsu | Fujitsu BX960 S1 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 40.49@28 tiles View Disclosure |
2 sockets 16 total cores 32 total threads |
10/19/10 |
Cisco | Cisco UCS B230 M1 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 39.19@27 tiles View Disclosure |
2 sockets 16 total cores 32 total threads |
10/19/10 |
HP | HP ProLiant BL620c G7 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 37.92@28 tiles View Disclosure |
2 sockets 16 total cores 32 total threads |
10/19/10 |
Dell | Dell PowerEdge R810 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 37.28@26 tiles View Disclosure |
2 sockets 16 total cores 32 total threads |
07/27/10 |
Dell | Dell PowerEdge M910 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 37.11@26 tiles View Disclosure |
2 sockets 16 total cores 32 total threads |
07/13/10 |
Fujitsu | Fujitsu BX960 S1 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 32.82@22 tiles View Disclosure |
4 sockets 16 total cores 32 total threads |
06/29/10 |
Dell | Dell PowerEdge M905 VMware ESX v4.0 |
VMmark v1.1.1 22.90@17 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
06/19/09 |
Dell | Dell PowerEdge R905 VMware ESX v4.0 |
VMmark v1.1 22.70@16 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
05/19/09 |
HP | HP ProLiant DL585 G5 VMware ESX v4.0 |
VMmark v1.1 22.11@15 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
04/24/09 |
HP | HP ProLiant BL685 G6 VMware ESX v4.0 |
VMmark v1.1 20.87@14 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
04/24/09 |
HP | HP ProLiant DL585 G5 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 20.43@14 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
01/27/09 |
Dell | Dell PowerEdge R905 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 20.35@14 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
11/12/08 |
HP | HP ProLiant BL685c G5 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 19.96@14 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
12/09/08 |
Dell | Dell PowerEdge M905 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 19.91@14 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
11/12/08 |
IBM | IBM Bladecenter LS42 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 19.17@14 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
01/13/09 |
IBM | IBM BladeCenter LS42 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 16.81@11 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
11/12/08 |
Dell | Dell PowerEdge R905 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 15.35@11 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
10/08/08 |
Dell | Dell PowerEdge M905 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 15.09@11 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
10/01/08 |
Dell | Dell PowerEdge R905 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 14.84@10 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
09/17/08 |
HP | HP ProLiant DL585 G5 VMware ESX v3.5.0 Update 1 |
VMmark v1.1 14.74@10 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
08/05/08 |
Dell | Dell PowerEdge M905 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 14.28@11 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
08/12/08 |
Dell | Dell PowerEdge R905 VMware ESX v3.5.0 Update 1 |
VMmark v1.0 14.17@10 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
05/06/08 |
HP | HP ProLiant DL580 G5 VMware ESX v3.5.0 Update 1 |
VMmark v1.1 14.14@10 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
09/05/08 |
Dell | Dell PowerEdge R900 VMware ESX v3.5.0 Update 1 |
VMmark v1.0 14.05@10 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
07/08/08 |
IBM | IBM System x3850 M2 VMware ESX v3.5.0 |
VMmark v1.0 13.16@9 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
03/26/08 |
Sun | Sun Fire X4450 VMware ESX v3.5.0 |
VMmark v1.0 12.23 @ 8 Tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
04/25/08 |
Dell | Dell PowerEdge R900 VMware ESX v3.5.0 |
VMmark v1.0 12.23@8 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
11/19/07 |
HP | HP ProLiant DL580 G5 VMware ESX v3.0.2 |
VMmark v1.0 11.54@8 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
08/31/07 |
HP | HP ProLiant BL680 G5 VMware ESX v3.0.2 |
VMmark v1.0 10.17@7 tiles View Disclosure |
4 sockets 16 total cores 16 total threads |
08/31/07 |
12 Cores
Submitter | System Description | VMmark Version & Score | Processors | Published Date |
---|---|---|---|---|
Fujitsu | Fujitsu BX924 S2 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 40.86@30 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
10/19/10 |
HP | HP ProLiant DL380 G7 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 38.97@28 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
10/19/10 |
Dell | Dell PowerEdge R710 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 38.39@27 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
10/05/10 |
Fujitsu | Fujitsu BX924 S2 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 38.39@27 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
09/07/10 |
Dell | Dell PowerEdge M610x VMware ESX v4.0 Update 2 |
VMmark v1.1.1 38.38@27 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
10/19/10 |
Cisco | Cisco UCS B250 M2 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 38.04@27 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
10/19/10 |
SGI | SGI C2005-TY15 VMware ESX v4.1 |
VMmark v1.1.1 36.76@27 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
10/05/10 |
Cisco | Cisco UCS B250 M2 VMware ESX v4.0 Update 1 |
VMmark v1.1.1 35.83@26 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
04/06/10 |
Fujitsu | Fujitsu RX200 S6 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 35.09@27 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
08/10/10 |
Fujitsu | Fujitsu RX300 S6 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 35.07@27 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
08/10/10 |
Dell | Dell PowerEdge M610x VMware ESX v4.0 Update 2 |
VMmark v1.1.1 34.80@27 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
10/05/10 |
Dell | Dell PowerEdge R710 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 34.78@28 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
10/05/10 |
Dell | Dell PowerEdge R710 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 33.64@25 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
09/07/10 |
Fujitsu | Fujitsu BX922 S2 VMware ESX v4.0 Update 1 |
VMmark v1.1.1 32.89@24 tiles View Disclosure |
2 sockets 12 total cores 24 total threads |
04/06/10 |
HP | HP ProLiant DL385 G6 VMware ESX v4.0 |
VMmark v1.1 15.54@11 tiles View Disclosure |
2 sockets 12 total cores 12 total threads |
06/02/09 |
8 Cores
Submitter | System Description | VMmark Version & Score | Processors | Published Date |
---|---|---|---|---|
Fujitsu | Fujitsu BX924 S2 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 30.05@20 tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
10/19/10 |
HP | HP ProLiant DL380 G7 VMware ESX v4.0 Update 2 |
VMmark v1.1.1 29.46@20 tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
10/19/10 |
Fujitsu | Fujitsu RX300 S6 VMware ESX v4.0 Update 1 |
VMmark v1.1.1 28.02@18tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
05/04/10 |
Fujitsu | Fujitsu BX922 S2 VMware ESX v4.0 Update 1 |
VMmark v1.1.1 27.99@18tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
05/04/10 |
SGI | SGI C2104-TY3 VMware ESX v4.1 |
VMmark v1.1.1 25.67@18tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
09/21/10 |
HP | HP ProLiant ML370 G6 VMware ESX v4.0 Update 1 |
VMmark v1.1.1 25.29@18tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
04/06/10 |
HP | HP ProLiant BL490c G6 VMware ESX v4.0 |
VMmark v1.1.1 25.27@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
04/20/10 |
Fujitsu | Fujitsu RX300 S5 VMware ESX v4.0 |
VMmark v1.1.1 25.16@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
11/09/09 |
Cisco | Cisco UCS B200 M1 VMware ESX v4.0 |
VMmark v1.1.1 25.06@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
01/12/10 |
SGI | SGI XE500 VMware ESX v4.0 Update 1 |
VMmark v1.1.1 24.56@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
03/09/10 |
HP | HP ProLiant BL490c G6 VMware ESX v4.0 |
VMmark v1.1.1 24.54@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
09/22/09 |
SGI | SGI XE270 VMware ESX v4.0 Update 1 |
VMmark v1.1.1 24.52@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
04/06/10 |
Lenovo | Lenovo R525 G2 VMware ESX v4.0 |
VMmark v1.1.1 24.35@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
06/30/09 |
Dell | Dell PowerEdge R710 VMware ESX v4.0 |
VMmark v1.1.1 24.27@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
09/08/09 |
HP | HP ProLiant BL490 G6 VMware ESX v4.0 |
VMmark v1.1 24.24@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
05/19/09 |
Fujitsu | Fujitsu RX200 S5 VMware ESX v4.0 |
VMmark v1.1.1 24.20@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
08/11/09 |
Sun | Sun Fire X4270 VMware ESX v4.0 |
VMmark v1.1.1 24.18@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
09/28/09 |
HP | HP ProLiant DL380 G6 VMware ESX v4.0 |
VMmark v1.1 24.15@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
05/19/09 |
Cisco | Cisco B200-M1 VMware ESX v4.0 |
VMmark v1.1 24.14@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
04/21/09 |
IBM | IBM Bladecenter HS22 VMware ESX v4.0 |
VMmark v1.1.1 24.05@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
06/30/09 |
Dell | Dell PowerEdge R710 VMware ESX v4.0 |
VMmark v1.1 24.00@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
04/21/09 |
HP | HP ProLiant DL370 G6 VMware ESX v4.0 |
VMmark v1.1 23.96@16tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
03/30/09 |
Dell | Dell PowerEdge M610 VMware ESX v4.0 |
VMmark v1.1 23.90@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
04/21/09 |
IBM | IBM System x3650 M2 VMware ESX v4.0 |
VMmark v1.1 23.89@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
04/27/09 |
Dell | Dell PowerEdge R710 VMware ESX v4.0 |
VMmark v1.1 23.55@16tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
03/30/09 |
Inspur | Inspur NF5280 VMware ESX v4.0 |
VMmark v1.1 23.45@17tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
03/30/09 |
Intel | Supermicro 6026-NTR+ VMware ESX v3.5.0 Update 4 |
VMmark v1.1 14.22@10 tiles View Disclosure |
2 sockets 8 total cores 16 total threads |
03/30/09 |
HP | HP ProLiant385G5p VMware ESX v3.5.0 Update 3 |
VMmark v1.1 11.28@8 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
12/30/08 |
HP | HP ProLiant BL495cG5 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 11.23@8 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
12/30/08 |
Dell | Dell PowerEdge R805 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 11.22@8 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
11/12/08 |
Dell | Dell PowerEdge M805 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 11.00@8 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
11/17/08 |
HP | HP ProLiant ML370 G5 VMware ESX v3.5.0 Update 3 |
VMmark v1.1 9.15@7 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
10/09/08 |
Dell | Dell PowerEdge M600 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 8.97@6 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
09/24/08 |
Dell | Dell PowerEdge 2950 III VMware ESX v3.5.0 Update 2 |
VMmark v1.1 8.94@6 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
09/29/08 |
HP |
HP ProLiant ML370 G5 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 8.66@6 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
10/06/08 |
IBM | IBM System x3650 VMware ESX v3.5.0 Update 1 |
VMmark v1.1 8.63@6 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
09/02/08 |
Dell | Dell PowerEdge 2950 III VMware ESX v3.5.0 |
VMmark v1.0 8.47@6 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
11/06/07 |
IBM | IBM System BladeCenter HS21 XM VMware ESX v3.5.0 Update 1 |
VMmark v1.1 8.44@6 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
09/05/08 |
Dell | Dell PowerEdge M600 VMware ESX v3.5.0 |
VMmark v1.0 8.37@6 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
02/08/08 |
Sun | Sun Fire X4240 VMware ESX v3.5.0 Update 1 |
VMmark v1.1 8.07@6 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
10/23/08 |
Dell | Dell PowerEdge R805 VMware ESX v3.5.0 Update 1 |
VMmark v1.0 7.96@6 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
05/06/08 |
Sun | Sun Fire X4240 VMware ESX v3.5.0 Update 1 |
VMmark v1.1 7.92@6 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
10/10/08 |
Dell | Dell PowerEdge M805 VMware ESX v3.5.0 Update 2 |
VMmark v1.1 7.88@6 tiles View Disclosure |
2 sockets 8 total cores 8 total threads |
08/12/08 |
Sun | Sun Blade X8440 VMware ESX v3.5.0 |
VMmark v1.0 7.37@5 tiles View Disclosure |
4 sockets 8 total cores 8 total threads |
02/04/08 |
Sun | Sun Blade X8440 VMware ESX v3.0.2 |
VMmark v1.0 7.02@5 tiles View Disclosure |
4 sockets 8 total cores 8 total threads |
1/08/08 |
View Historical VMmark Version 1 Results
View Withdrawn VMmark Version 1 Results
Fair Use Guidelines
- Only official VMmark metrics and submetrics may be used in comparisons.
- Any competitive comparison of VMmark metrics must also include a statement providing the basis for making the comparison and the date when the claim is first made. A published comparison could, for example, state the claim is based on having best VMmark result out of all results using four core systems, published as of the original publication date.
- Comparisons of VMmark metrics and submetrics to any other benchmark metrics are not allowed. VMmark utilizes other benchmarking software as load generators and produces results which are not comparable to the original benchmark metrics.
VMmark FAQs
Note: VMmark 1.x is now permanently retired. It will remain available for download for academic purposes only. The VMmark 1.x results page will remain available for reference.
Get an accurate measurement of application performance in virtualized environments with VMmark, the industry's first virtualization benchmark for x86-based computers. For academic use only.
What is VMmark?
VMmark is the first benchmark that was designed specifically to quantify and measure the performance of virtualized environments. It features a novel tile-based scheme for measuring the scalability of consolidated workloads and provides a consistent methodology that captures both the overall scalability and individual application performance. The VMmark benchmark is built on our expertise in virtualization performance and incorporates popular workloads from application categories most commonly represented in customer data centers.
Why is there a need for a new benchmark?
Traditional server benchmarks that exist today were developed with neither virtual machines nor server consolidation in mind and focus on a single workload per server. These benchmarks do not capture the system behavior induced by multiple virtual machines and fail to provide sufficient insight into the scalability of virtual environments supporting multiple simultaneous workloads on the same server. Organizations implementing or evaluating virtualization platforms today also need a more realistic and specialized benchmark to help them compare performance and scalability of different virtualization platforms, make appropriate hardware choices and measure platform performance on an ongoing basis. Clearly, a more sophisticated approach is required to quantify a virtualization environment's performance and develop meaningful and precise metrics in order to effectively compare the suitability and performance of different hardware platforms for virtual environments. Also, there is a need for a common workload and methodology for virtualized systems so that benchmark results can be compared across different virtualization platforms.
What are some specific requirements for developing such a benchmark?
Besides the need to capture key performance characteristics of virtual systems, an appropriate virtual machine benchmark must employ realistic, diverse workloads running on multiple operating systems. Further, there is a need to define a single, easy to understand metric while ensuring that the benchmark is representative of various end user environments. The benchmark specification needs to be platform neutral and should also provide a methodical way to measure scalability so that the same benchmark can be used for small servers as well as larger servers from different hardware vendors.
Why did VMware develop VMmark?
VMware realized the need for a virtualization benchmark early on with more and more customers asking for metrics to compare different hardware platforms and configurations on which to run their virtualized environments. VMmark provides a standardized way to compare platforms that customers have come to expect from enterprise software.
Will VMmark be an industry standard? If so, what is VMware doing towards this goal?
VMware is actively working on open standards on virtualization benchmarks. In October 2006, SPEC formed a working group to develop a standard benchmark for measuring virtualization performance. This working group was formed at the request of VMware. By March 2007 the working group had agreed on the design goals and project plan, and was able to graduate to a subcommittee. Paula Smith of VMware was the chairperson of the working group and continues to chair the subcommittee. We are an active participant in the subcommittee, along with many of our major partners and a few of our competitors. Current participants include: AMD, Dell, Fujitsu Siemens, Hewlett-Packard, Intel, IBM, Microsoft, Red Hat, Sun Microsystems, SWsoft and VMware. Additional information on the subcommittee can be found at: http://www.spec.org/specvirtualization/
How was VMmark developed?
Nearly two years of engineering effort has gone into the design and implementation of the benchmark culminating in a private beta release in December, 2006 and the current public beta (launched in July 2007) as a part of VMware’s normal product release cycle. In the effort to build a reliable and robust benchmark that truly represents customer environments, VMware has taken into account extensive survey data from its customers to understand what types of applications and configurations are typically run in virtualized environments. VMware has also worked closely with its partners to design and implement the benchmark across various software and hardware platforms. Throughout the course of the benchmark development, VMware has also evaluated numerous workloads and run hundreds of experiments to make sure that the benchmark is reliable and robust.
What is a tile?
A tile is a collection of six diverse workloads concurrently executing specific software. Running on one of two separate operating systems, each workload runs in its own virtual machine and executes applications found in all the world's datacenters. Included in a single tile are a web server, file server, mail server, database, java server, as well as an idle machine. Each virtual machine in a tile is tuned to use only a fraction of the system's total resources. As a tile, the aggregate of all six workloads normally utilizes less than the full capacity of modern servers. Therefore, the complete saturation of a system's resources and accurate measurement of server performance with VMmark require the execution of multiple tiles simultaneously.
How does VMmark work?
VMmark is designed as a tile-based benchmark consisting of a diverse set of workloads commonly found in the datacenter, including database server, file server, web server, and Java server. The workloads comprising each tile are run simultaneously in separate virtual machines at load levels that are typical of virtualized environments. The performance of each workload is measured and then combined with the other workloads to form the score for the individual tile. Multiple tiles can be run simultaneously to increase the overall score. This approach allows smaller increases in system performance to be reflected by increased scores in a single tile and larger gains in system capacity to be captured to adding additional tiles. (Future work will present data to demonstrate the ability of multiple tiles to measure performance of larger multiprocessor systems using a well-defined reference score). Each workload within a VMmark tile is constrained to execute at less than full utilization of its virtual machine. However, the performance of each workload can vary to a degree with the speed and capabilities of the underlying system. For instance, disk-centric workloads might respond to the addition of a fast disk array with a more favorable score. These variations can capture system improvements that do not warrant the addition of another tile. However, the workload throttling will force the use of additional tiles for large jumps in system performance. When the number of` tiles is increased, workloads in existing tiles might measure lower performance. However, if the system has not been overcommitted, the aggregate score, including the new tile, should increase. The result is a flexible benchmark metric that provides a relative measure of the number of workloads that can be supported by a particular system as well as the overall performance level within the virtual machines.
Who will use VMmark?
VMmark was developed as a useful tool for hardware vendors, system integrators, and customers to evaluate the performance of their systems. Many customers will not run the benchmark themselves, but rather rely on published VMmark scores from their hardware vendors to make purchasing and configuration decisions for their virtualization infrastructure.
What are the use cases for VMmark?
The main use-case for VMmark is to compare the performance of different hardware platforms and configurations. Organizations implementing or evaluating virtualization platforms today will use VMmark for comparing performance and scalability of different virtualization platforms, making appropriate hardware choices and for measure platform performance on an ongoing basis. It is also important to note that VMmark is neither a capacity planning tool nor a sizing tool. It does not provide deployment guidelines for specific applications. Rather VMmark is meant to be representative of a general-purpose virtualization environment. The virtual machine configurations and the software stacks inside the virtual machines are fixed as part of the benchmark specification. Recommendations derived from VMmark results will capture many common cases; however, specialized scenarios will likely require individual measurement.
What are the benefits of VMmark?
With VMmark, organizations now have a robust and reliable and benchmark that captures the key performance characteristics of virtual systems; is representative of end user environments running multiple workloads; is platform neutral and provides a methodical way to measure scalability so that the same benchmark can be used across different hardware platforms. With VMmark, organizations now finally have a virtualization benchmark that works. With VMmark, organizations can compare performance and scalability of different virtualization platforms, make appropriate hardware choices and monitor virtual machine performance on an ongoing basis.
How do I interpret a VMmark score?
A VMmark score is a measure of the performance of both the hardware and virtualization layers of a virtualization platform. Each score represents the performance relative to a fixed reference platform. Though the reference platform is from a previous hardware generation, making comparisons between it and newer systems not very meaningful, its use allows for easy comparisons between various contemporary platforms and configurations. A score is obtained by measuring the aggregate throughput achieved by multiple workloads executing simultaneously on the virtualization platform. A set of six specific workloads, each in its own virtual machine, are run for a specific length of time. These six workload virtual machines are collectively defined as a VMmark tile. During a VMmark run each individual workload generates a raw throughput metric -- for example, the throughput of the database workload is measured in transactions per minute. Upon completion of a run each of these raw metrics is normalized with respect to the reference platform then the geometric mean of the normalized individual scores is computed. The resulting score is a measure of the throughput of the tested platform relative to the reference platform. In addition to this score, each VMmark result also includes the number of VMmark tiles used in the benchmark run. With increasing system resources (for example, more CPU cores) multiple VMmark tiles (that is, complete sets of the six workload virtual machines) can be run simultaneously in order to fully utilize a virtualization platform. After calculating the score for each tile the individual tile scores are added together to produce the VMmark score. A VMmark full disclosure report also includes the raw and normalized results for each underlying workload as well as complete details of the virtualization platform configuration. In some cases, studying the workload metrics along with the platform configuration can provide insight into system performance and scaling. For a more detailed description of the benchmark scoring methodology see the VMmark Benchmarking Guide.
How do I compare VMmark scores across different virtualization platforms?
A higher VMmark score implies that a virtualization platform is capable of sustaining greater throughput in a mixed workload consolidation environment. A larger number of VMmark tiles used to generate the benchmark means that the platform supported more virtual machines during the benchmark run. Typically, a higher benchmark score requires a higher number of tiles. If two different virtualization platforms achieve similar VMmark scores with a different number of tiles, the score with the lower tile count is generally preferred. The higher tile count could be a sign that the underlying hardware resources were not properly balanced. Studying the individual workload metrics is suggested in these cases.
How is VMmark version 1.1 different from version 1.0?
In order to address the growing prevalence of 64bit applications and OSs, in VMmark 1.1 the Java server, database server, and web server workloads within the tile are 64-bit. The mail server, file server, and standby server remain 32-bit and unchanged from VMmark 1.0.
Performance Blog
Learn about the performance of virtualized environments at the VROOM! Blog
Read Now