Certification Test Matrix

Goals:

Ensure compliance with virtual appliance best practices
Ensure compatibility across VMware product line
Ensure ease-of-use for initial customer experience with appliance

Test Matrix:

ID	Test	Details	Required	Notes
1	VMware ESX version of the virtual appliance	Appliance vendor must provide a version of their appliance that has been converted to VMware ESX format	Yes	Exceptions may be granted on a case-by-case basis to allow vendors to offer only one file to download. These cases will require explicit documentation for how a user may convert a VMware Server/VMware Workstation/VMware Player formatted virtual appliance to VMware ESX. The vendor must test these procedures before an exception will be granted.
2	VMware Server version of the virtual appliance	Appliance vendor should provide a version of their appliance that has virtual disks in the VMware Server/VMware Workstation/VMware Player format	No	While not a requirement, most appliance vendors start by developing a virtual appliance on a hosted-class product such as VMware Workstation or VMware Server, and then convert the appliance for use on hypervisor-class products (VMware ESX and ESXi).
3	Documentation	The virtual appliance package includes sufficient documentation. Documentation may all be included within the product (command line help, web console help, etc) or as a separate file. Ideally, the package should include a Getting Started Guide or Read Me file to get the user started (with default user ids, passwords, etc.). A Getting Started Guide should describe common usage scenarios and should include a sample that users can follow for their first pass at using the virtual appliance	Yes	Subjective to some extent. Does the appliance provide a good out-of-the-box experience for a new user?
4	VMware ESX Installation Instructions	Written documentation on how to run the virtual appliance on VMware ESX. Documentation should include: The process for copying the VMware ESX formatted VMDK file(s) to VMware ESX The process for creating a new VM on VMware ESX and using the existing VMDK file (added to VMware ESX in step 1) as the virtual disk The optimal settings for the virtual machine when running in production mode on VMware ESX (ideal memory allocation, network settings, etc.)	Yes
5	VMware Server Installation Instructions	In the case that production use will be supported in a hosted environment, the virtual appliance documentation should include instructions for ensuring that the virtual appliance has been configured optimally (ideal memory allocation, network settings, etc.)	No
6	SCSI Virtual Disk(s)	Open vmx file and verify that virtual disks are setup as SCSI disks and not IDE disks	Yes	Should not be an issue if vendor provides a VMware ESX version of their appliance
7	LSI Logic Driver	The vmx file should be set with scsi0.virtualDev = "lsilogic"	No	Best practice for most Linux distributions. See the VMware Guest Operating System Installation Guide for more detail.
8	Disks split into 2GB files	Virtual disks should be split into 2 GB files to ensure compatibility with file systems that to not support large files (FAT32)	No
9	Disks pre-allocated	Virtual disks should be fully allocated at design time (do not use "grow as you go" disks)	No
10	Memory matches docs	The memsize setting for the appliance should match the documented minimum memory required for the appliance	Yes
11	No non-standard configuration settings	The vmx file should not contain any undocumented settings	Yes
12	No Snapshots	Verify that the virtual appliance does not include a snapshot	Yes
13	Snapshots disabled	The vmx file should be set with snapshot.disabled = "TRUE"	No
14	VMware Tools Installed	VMware Tools should be installed in the virtual appliance. Only in extreme cases will an exception be granted.	Yes
15	Unused hardware devices removed	Remove unused devices from the virtual machine definition. Typically servers do not need to include floppy drives, USB controller, Audio, etc.	No
16	Floppy not present	The vmx file should be set with floppy0.startConnected = "FALSE"	No
17	Devices set to auto-detect	The vmx file should be set with device.autodetect = "TRUE" for those devices which are included in the virtual machine definition	No
18	Anti-virus installed in Guest OS	Some level of security should be provided to ensure that users of an appliance cannot transmit files containing viruses	No	In some cases this will not be necessary as the appliance will be "read only" with the exception of non-executable log files and configuration files.
19	Performance benchmark	Provide guidelines around expected performance on a non-loaded VMware ESX running	No	This can provide expectations of performance for minimal hardware performance characteristics.
20	Layered application requirements	Does this virtual appliance require another VM such as a database or web front end? If so, what are the details?		As we build more and more virtual appliances, there may come a need to create pods of virtual appliances. Each pod should be self contained. These can represent security pods, disaster recovery pods, etc...
Overall