Using Storage Area Networks with ESX Server

VMware ESX Server 2.0

Features | Documentation | Knowledge Base | Discussion Forums

VMware ESX Server can be used effectively with storage area networks (SANs). ESX Server supports Qlogic and Emulex host bus adapters, which allow an ESX Server computer to be connected to a SAN and to see the disk arrays on the SAN.

The SCSI configuration information contained in this section also applies to Fibre Channel adapters, but note that FC adapters may require additional configuration as well.

For information on supported SAN hardware, download the VMware ESX Server SAN Compatibility List from the VMware Web site at www.vmware.com/support/esx2.

Understanding Storage Arrays

Large storage systems (also known as disk arrays) combine numerous disks into arrays for availability and performance. Typically, a collection of disks is grouped into a Redundant Array of Inexpensive Disks (RAID) array to protect the data by eliminating disk drives as a potential single point of failure.

Disk arrays carve the storage RAID set into logical units (LUNs) that are presented to the server in a manner similar to an independent single disk. Typically, LUNs are few in number, relatively large, and fixed in size.

You can create LUNs with the storage management application of your disk array.

Installing ESX Server with Attached SANs

We recommend that you install ESX Server software on a local SCSI disk.

Caution: Be sure that all Fibre Channel adapters (QLogic or Emulex) are detached from the SAN during ESX Server installation. If they are not disconnected, SAN disks are often displayed first in the list of drives displayed during installation. This may cause confusion when the service console is being installed.

In addition, we recommend that all Fibre Channel adapters are dedicated exclusively for the virtual machines. Even though these FC adapters are dedicated to virtual machines, the LUNs on the SANs are visible to system management agents on the service console.

Configuring VMFS Volumes on SANs

Be sure that only one ESX Server has access to the SAN while you are using the VMware Management Interface to configure the SAN and format the VMFS-2 volumes. After you have finished the configuration, be sure that all partitions on the physically shared SAN disk are set for public or shared access for access by multiple ESX Servers (see VMFS Accessibility).

For information on configuring SANs, scanning for LUNs and setting persistent bindings through the VMware Management Interface, see Configuring a Swap File.

Scanning for Devices and LUNs

ESX Server scans for devices, and LUNs on these devices, whenever a Fibre Channel driver is loaded. You can manually initiate a scan through the VMware Management Interface or by using the vmkfstools -s command.

You may want to rescan devices or LUNs whenever you add a new disk array to the SAN or create new LUNs on a disk array. You may also want to rescan LUNs when you change the LUN masking on a disk array.

Note: For QLogic adapters, you must do one additional step before scanning, in order to recognize new LUNs on existing disk arrays. For each and every numeric entries in /proc/scsi/qla2200 or /proc/scsi/qla2300, run the appropriate command, as the root user, on the service console:

echo "scsi-qlascan" > /proc/scsi/qla2200/<numeric_entry>

echo "scsi-qlascan" > /proc/scsi/qla2300/<numeric_entry>

These commands tell the QLogic driver to clear its cache of existing LUNs.

Caution: Do not run the vmkfstools -s command on non-Fibre Channel adapters.

Type the following command to scan for devices or LUNs through a particular FC SCSI adapter:

vmkfstools -s <FC_SCSI_adapter>

For example, if you wanted to rescan SCSI adapter vmhba0 for new (and removed) devices or LUNs, type:

vmkfstools -s vmhba0

For more information on using vmkfstools, see Using vmkfstools.

Note: If you are using multipathing with multiple FC HBAs, then you should run this command on all of the FC HBAs. If, after your rescan, you see new LUNs and they have VMFS volumes, then you will see the appropriate subdirectories when you view the contents of the /vmfs directory.

Choosing QLogic Adapters

If you are using a QLogic storage adapter, then be sure to choose the right driver version:

IBM storage — QLogic driver version 6.04 (default)
HP storage — QLogic driver version 6.04 (default)
EMC storage — QLogic driver version 6.04 (default)

Changing VMkernel Configuration Options for SANs

In order to use all storage devices on your SAN, you may need to change some VMkernel configuration options as described below.

To make these changes, complete the following steps.

Log in to the VMware Management Interface as root.
The Status Monitor page appears.
Click the Options tab.
Click Advanced Settings.
To change an option, click the current value, then enter the new value in the dialog box and click OK.
For more information on changing these settings, see Changing Advanced Settings.

Detecting All LUNs

By default, the VMkernel scans for only LUN 0 to LUN 7 for every target. If you are using LUN numbers larger than 7 you must change the setting for the DiskMaxLUN field from the default of 8 to the value that you need. For example, if you now have LUN numbers 0 to 15 active, set this option to 16. Currently, an ESX Server machine can see a maximum of 128 LUNs over all disk arrays on a SAN.

By default, the VMkernel is configured to support sparse LUNs — that is, a case where some LUNs in the range 0 to N-1are not present, but LUN N is present. If you do not need to use such a configuration, you can change the DiskSupportSparseLUN field to 0. This change decreases the time needed to scan for LUNs.

The DiskMaskLUNs configuration option allows the masking of specific LUNs on specific HBAs. Masked LUNs are not touched or accessible by the VMkernel, even during initial scanning. The DiskMaskLUNs option takes a string comprised of the adapter name, target ID and comma-separated range list of LUNs to mask. The format is as follows:

<adapter>:<target>:<comma_separated_LUN_range_list>;

For example, you want to mask LUNs 4, 12, and 54-65 on vmhba 1 target 5, and LUNs 3-12, 15, and 17-19 on vmhba 3 target 2. To accomplish this, set the DiskMaskLUNs option to the following:

"vmhba1:5:4,12,54-65;vmhba3:2:3-12,15,17-19;"

Note: LUN 0 cannot be masked.

The DiskMaskLUNs option subsumes the DiskMaxLUN option for adapters that have a LUN mask. In other words, continuing the preceding example, there are four adapters, vmhba0, vmhba1, vmhba2, and vmhba3, and the DiskMaxLUN option is set to 8. In this example, vmhba0 and vmhba2 only scan LUNs 0-7, but vmhba1 and vmhba3 scan all LUNs that are not masked, up to LUN 255, or the maximum LUN setting reported by the adapter, whichever is less.

For administrative or security purposes, you can use LUN masking to prevent the server from seeing LUNs that it doesn't need to access. Refer to your documentation on disk arrays for more information.

Using IBM FAStT Disk Arrays

An IBM FAStT disk array sometimes returns vendor-specific status codes that ESX Server interprets as errors. These status codes are temporary -- indicating, for example, that the firmware has been upgraded or that the battery for the disk cache needs to be charged. ESX Server, in its default configuration, may interpret these status codes to mean that a LUN exists but is not accessible.

You avoid this problem by using a special ESX Server 2.0 configuration option. Log in to the management interface as the root user, click Advanced Settings, then click VMkernel Configuration. Find the option DiskRetryUnitAttention and be sure that it is enabled (the default).

With this option enabled, ESX Server automatically retries SCSI commands when these vendor-specific status codes are received.

Troubleshooting SAN Issues with ESX Server

You can view LUNs through the VMware Management Interface or viewing the output of ls /proc/vmware/scsi/<FC_SCSI_adapter>. If the output differs from what you expect, then check the following:

DiskMaxLUN — the maximum number of LUNs per vmhba that are scanned by ESX Server.
You can view and set this option through the VMware Management Interface (Advanced Settings in the Options page) or by viewing this setting through/proc/vmware/config.
DiskSupportSparseLUN — if this option is on, then ESX Server scans past any missing LUNs. If this option is off, ESX Server stops scanning for LUNs if any LUN is missing.
You can view and set this option through the VMware Management Interface (Advanced Settings in the Options page) or by viewing this setting through/proc/vmware/config.
LUN masking — With LUN masking, each LUN is exclusively assigned and accessed by a specific list of connections. Be sure that LUN masking is implemented properly and that the LUNs are visible to the HBAs on ESX Server.
Zoning — Zoning limits access to specific storage devices and increases security and decreases traffic over the network. If you use zoning, be sure that zoning on the SAN switch is set up properly and that all vmhba and the controllers of the disk array are in the same zone.
Storage controller — If a disk array has more than one storage controller, then make sure that the SAN switch has a connection to the controller that owns the LUNs you wish to access. On some disk arrays, only one controller is "active" and the other controller is "passive" until there is a failure. If you are connected to the wrong controller, then you may not see the expected LUNs, or you may see the LUNs, but may get errors when trying to access them.

For more information on using SANs with ESX Server, be sure to check the Knowledge Base on the VMware Web site at www.vmware.com/support/.

Prev Contents Last Next

VMware ESX Server 2.0

Support Resources