VMware

VDDK 5.1 Release Notes

Release date: 10 SEP 2012 | Builds: ESXi 799733, VDDK 774844.
Last document update: 16 APR 2013.
Check frequently for additions and updates to these release notes.

Contents


About the 5.1 Virtual Disk Development Kit

The Virtual Disk Development Kit (VDDK) 5.1 is an update to support vSphere 5.1 and to resolve issues discovered in previous releases. VDDK 5.1 adds support for ESXi 5.1 and vCenter Server 5.1, and was tested with the following VMware platform products:

  • ESXi 5.1, ESXi 5.0, and ESX/ESXi 4.1
  • vCenter Server 5.1 and 5.0 managing ESX/ESXi 4.1 and later

VDDK 5.1 was tested with (and supports) the following operating systems to perform proxy backup:

  • Windows Server 2012 (64-bit only)
  • Windows 8
  • Windows Server 2008 R2 (64-bit only)
  • Windows Server 2003 R2
  • Windows XP
  • SUSE Linux Enterprise Server SLES 11 SP2.
  • Red Hat Enterprise Linux RHEL 6.1.

For Linux, only the default out-of-box file system is tested and supported.

The handling of writable snapshots changes in the ESXi 5.1 release, however this seldom affects backup applications, which can use read-only snapshots.

VMware Consolidated Backup (VCB) has a writeup (knowledge base article) showing the support matrix for storage multipathing. VMware does not provide a similar support matrix for VDDK. Customers should seek this information from their backup software vendors.

These release notes provide information about issues identified or resolved since VDDK 5.0. For information about the 5.0 release, see the VDDK 5.0 Release Notes. For general information about the development kit, how to obtain and install the software, redistributing it, and issues in previous releases, see the VDDK documentation landing page.

Compatibility Notices for Partners

Linux proxy SAN backup can fail with powered on VM. If any virtual machine I/O operations are occurring for a VMDK on a LUN when the Linux proxy starts SAN transport backup, proxy systems before SLES 11.2 and RHEL 6.2 receive an EIO instead of EBUSY error, causing backup to fail. This is more likely to occur with a powered on VM. The cause is a Linux bug in handling SCSI reservation conflicts, encountered on the following releases, and fixed in newer kernels:

  • SLES 10.4 and SLES 11.1
  • RHEL 5.8, RHEL 5.9, RHEL 6.0, RHEL 6.1

Only one plug-in. After implementation of SSL certificate checking, it was necessary to limit the VDDK framework to one plug-in, diskLibPlugin.dll or libdiskLibPlugin.so, which handle the advanced transport modes: SAN, HotAdd, and NBDSSL. As a result, VDDK 5.1 can no longer load a custom partner plug-in. This is a known issue, and it will be corrected in a future release.

Registry on 64-bit Windows. To support registry redirection, registry entries needed by VDDK on 64-bit Windows must be placed under registry hierarchy Wow6432Node. This is the correct location for both 32-bit and 64-bit binaries on 64-bit Windows. On 32-bit Windows, VDDK registry entries are in the old location.

Snapshot changes. In previous VMware releases, reverting to a writable snapshot sometimes left orphaned virtual disks that the system never removed. In this release, writable snapshots are correctly accounted for as sibling snapshots. This avoids orphaned disks, and streamlines snapshot management because the disk chain always matches the snapshot hierarchy. VMware snapshots are read-only by default. In a CreateSnapshot call, the quiesce flag synchronizes data and creates a read/write snapshot for use with Microsoft VSS, which requires writable snapshots. VMware recommends testing snapshot behavior for that use case.

Licensing. In vSphere 5.1, the SCSI HotAdd feature is always available. In vSphere 5.0, SCSI HotAdd was enabled only for vSphere editions Enterprise and higher, which have Hot Add licensing enabled. No separate Hot Add license was available for purchase as an add-on. Therefore, customers who used backup products (including VMware Data Recovery) with vSphere Essentials or Standard edition were not able to perform proxy-based backup.

SSL certificate check. VDDK 5.1 can be set to check the SSL certificate on Windows, or to verify the SSL thumbprint on Linux. To enable and disable this feature, you can edit the Windows registry, or add linuxSSL.verifyCertificates=1 to the configFile passed as a parameter to VixDiskLib_InitEx().

Relocated files. For reasons related to SSL security, in VDDK 5.1 the following directories found in previous releases were eliminated, and files therein moved to the following locations:

Windows VDDK\install\directory\plugins VDDK\install\directory\bin
Linux 32-bit VDDK/install/directory/plugins32 VDDK/install/directory/lib32
Linux 64-bit VDDK/install/directory/plugins64 VDDK/install/directory/lib64

Older VDDK 5.0 items follow below.

DNS. To avoid SSL certificate errors, DNS services must be configured in the backup proxy, consistent with vCenter Server and ESXi hosts in the datacenter.

Storage vMotion. To prevent orphaned virtual disk caused by Storage vMotion during backup, VDDK 5.0 added the VixDiskLib_PrepareForAccess and VixDiskLib_EndAccess functions, which disable and re-enable the RelocateVM_Task method. Storage vMotion can wait for the backup to finish, or relocate a different VM. See the Virtual Disk API Programming Guide for details.

Silent VDDK installation and the EULA. If you want to accept all defaults and install VDDK components silently on a Linux machine, providing you agree to terms in the end user license agreement (EULA), you can use the combined options --default EULA_AGREED=yes.

Windows silent install. If you want to perform silent install or uninstall of VDDK with InstallShield on Windows, run one of the following commands. The /s option means silent, the /x option means uninstall, and the /v option passes /qb (basic UI) or /qn (no UI) to MsiExec.

VMware-vix-disklib-5.0.0-317664.i386.exe /s /v"/qb"
VMware-vix-disklib-5.0.0-317664.i386.exe /s /x /v"/qn"

Recently Resolved Issues

The VDDK 5.1 release resolves the following issues:

  • Improved support for very large files.

    In previous VDDK releases, VixDiskLib_Create() enforced a 2GB limit on virtual disks, except with NTFS, FAT32, or FAT file systems, which were limited to 4GB. In this release, the NTFS limit is 16TB - 64KB (hex FFFFFFF0000), the FAT32 or FAT limit is still 4GB - 1 (hex FFFFFFFF), and the exFAT or ReFS limit is more than an exabyte (2^64 - 1). POSIX based file systems including NFS no longer have a 2GB file size limit. Although various checks are done to avoid creating impossibly large files, it becomes the customer's responsibility to cope with 2GB limits on NFSv2 or Linux kernel 2.4 and earlier.

  • Better read speed when using NBDSSL transport.

    Due to VDDK 5.0's use of OpenSSL library version 0.9.8q (also designated 0.9.8.17), throughput of NBDSSL transport was only about a third as fast as VDDK 1.2.1, or about one sixth the speed of unencrypted NBD transport. In VDDK 5.1, this was fixed by installing newer versions of bin\ssleay32.dll on Windows and libssl.so.* on Linux.

  • VixDiskLib transmits meaningful error code for thin-disk full.

    In earlier releases, VixDiskLib returned an “Unknown error” from diskLib when thin-provisioned disk ran out of space. The actual diskLib error is now returned with VIX_E_DISK_FULL, in the most-significant 16 bits of the error code.

  • QuickExit() error no longer appears.

    In VDDK 5.0, the warning “This application is not using QuickExit()” came from the vmacore library as a diagnostic for VMware applications. Though it was harmless, this message no longer appears.

  • VDDK returns error for SAN reads and writes to offline LUN.

    When attempting SAN transport reads or writes to virtual disk on an offline LUN, VDDK 5.0 I/O operations appeared to succeed despite the LUN being offline. The VixDiskLib_Open() function returned success but VixDiskLib_Read() and VixDiskLib_Write() failed to return an error even when they could not read or write the offline LUN. For backup and restore software running through a proxy, this issue caused backups and restores to return success, when data was never actually read from or written to virtual disk. This is fixed in the VDDK 5.1 and VDDK 5.0 U2 releases.

  • EndAccess always reenables relocation after PrepareForAccess.

    With vCenter Server managing vMotion on ESX/ESXii hosts, the VDDK 5.0 library routine VixDiskLib_PrepareForAccess() followed by VixDiskLib_EndAccess() occasionally did not change virtual-machine state to reenable relocation after backup. This seemed more likely to occur when VixDiskLib_PrepareForAccess() was called twice in a row (the second call got “[relocation] already disabled” response). Additionally the message “Insufficient permission in the host operating system” (error 3014, VIX_E_HOST_USER_PERMISSIONS) could appear after these calls. Now, processes running as Administrator or root should be able to change migration state.

  • Snapshot consolidation when cleaning up after HotAdd backup.

    When VMware Data Recovery (or VADP backup software) tried to clean up after HotAdd backup, snapshot delete failed as the appliance was reconfigured, and an exception was reported. The problem was caused by premature exit of the vpxa process, so vpxa did not inform backup code that HotAdd disks should be removed. Stale HotAdd disks remained, and were not recognized at subsequent backup time, causing failure of snapshot delete and buildup of redo logs. The solution is to wait longer for vpxa to report, and apply heuristics to clean up stale HotAdd disks if they appear. The VDDK 5.1 fix also works with earlier ESX/ESXi versions and for stale redo logs left by earlier VDDK versions.

  • HotAdd restore by Windows 2008 proxy when opening multiple disks.

    When a Windows Server 2008 virtual machine was deployed as a proxy, HotAdd failures occurred when writing to the second and subsequent disks of a virtual machine that was being restored. Due to SAN Policy changes, W2k8 R2 and W2k8 R2 SP1 were more prone to this than earlier editions. This advanced transport issue was fixed for the VDDK 5.1 release.

Known Issues and Workarounds

The following VDDK issues were reported after the October 2012 vSphere refresh.

  • With application level quiescing, Changed Block Tracking overstates changes.

    When calling QueryChangedDiskAreas to get changes between application level quiesced snapshots, the areas returned can represent a much larger amount of changes than the actual amount of changes. This issue occurs when taking quiesced snapshots to back up powered-on VMs running Windows Server 2003, 2008, 2008 R2, and 2012. The workaround is to set disk.enableUUID=false for any of these target VMs. To do this in the vSphere Client, power off the VM, click Edit Settings > Options > Advanced: General > Configuration Parameters, and change value of the disk.enableUUID parameter. This workaround results in file level quiesced backups instead of application level quiesced backups.

  • Prepare for Access did not throw the already-disabled fault.

    In vSphere 5.0 and 5.1, the VDDK libraries catch and ignore the VMODL_TYPE_VIM_FAULT_METHOD_ALREADY_DISABLED_FAULT return code when programs call VixDiskLib_PrepareForAccess() more than once before VixDiskLib_EndAccess(), or if a Storage vMotion is currently in progress. In the next major release, the VDDK libraries will transmit this return code.

  • Prepare for Access identity string length limitation.

    The VixDiskLib_PrepareForAccess() parameter *identity is a string limited to 50 characters.

  • Possible segmentation-violation if running multiple backup processes.

    When many simultaneous backup processes are running, some of them might crash with a SIGSEGV error. This is due to a possible race condition in VixDiskLib, which can be reduced by calling VixDiskLib_Exit() at end of your program. A fix has been identified, and will be available in the VDDK 5.1 first update release.

  • Connect call crashes if non-existent snapshot is specified.

    If a program calls VixDiskLib_ConnectEx() passing as third parameter the managed object reference (moRef) of a deleted or non-existent snapshot, the program crashes when dereferencing the snapshot moRef. A fix has been identified, and will be available in a future release. The workaround is to create a snapshot for each operation, and never operate on virtual machines with different backup programs at the same time.

  • SLES 10 SP2 requires newer OpenSSL library.

    VDDK 5.1 libraries are non-functional on SUSE Linux Enterprise Server 10.2, and perhaps other Linux distributions, until you install the libssl.so.0.9.8x shared object to replace the libssl.so.0.9.8t version. The newer OpenSSL library can deal with older versions of glibc on SLES 10. A fix has been identified, and will be available in the VDDK 5.1 first update release.

  • Occasional segmentation-violation crashes during library disconnect.

    When many Linux processes run simultaneously, some of them may get a SIGSEGV when calling VixDiskLib_Disconnect() before exit. This may also occur on Windows. One workaround is to have nothing important in your code after disconnect. A fix has been identified, and will be available in the VDDK 5.1 first update release.

  • Hang in connect or cleanup due to intermittent race condition.

    Windows processes can sometimes hang when calling VixDiskLib_ConnectEx() or VixDiskLib_Cleanup() due to a race condition while loading or unloading libraries. There is no known workaround. A fix has been identified, and will be available in the VDDK 5.1 first update release.

The following VDDK issues were discovered during the vSphere 5.1 release.

  • Disk Open fails without an SSL thumbprint from vCenter.

    Whenever vCenter Server does not provide an SSL thumbprint for the target host, the VDDK library might core dump in the function VixDiskLib_Open(). This could happen (but does not always happen, depending on memory content) when vCenter Server 5.1 manages virtual disk on ESXi 5.0 and older hosts. A fix has been identified, and will be available in the VDDK 5.1 first update release.

  • Specification of PluginsDir is misdocumented.

    Because of the Relocated files change described under Compatibility Notices for Partners above, in the Virtual Disk Programming Guide section Usage Message (pages 44-45), when using 32-bit Linux substitute /lib32 for /plugins32, and when using Windows substitute \bin for \plugins.

  • Problem in GetMetdataKeys with large number of snapshots.

    VDDK programs can crash in VixDiskLib_GetMetadataKeys() with an access violation during backup of a virtual machine with many snapshots (about 30). Although the VixDiskLibSample program does not do it, the workaround is to open, get metadata, and close each snapshot one at a time.

  • HotAdd open fails when volume spans multiple disks.

    When a Windows HotAdd proxy is configured with a volume that spans multiple disks, VixDiskLib_Open() fails to open the HotAdded virtual disk of the target virtual machine. A fix has been identified and will be available in a future release.

  • The vmware-mount command does not work correctly in this release.

    When you run the vmware-mount command in the VDDK 5.1 bin directory, it displays the following error: “Unable to mount the virtual disk. The disk may be in use... verify that the file is a valid virtual disk file.” This is the result of incompatible libraries. At this time VMware knows of no workaround.

  • Get Volume Handles might return zero for dynamic disk.

    When newly created dynamic volumes are not assigned to a drive letter, the VixMntapi_GetVolumeHandles() function returns zero volume handles for a Windows dynamic disk. A possible fix was identified and may be available in a future release. The workaround is to assign a drive letter.

  • With verbose logging, long-running VDDK processes might hang.

    The dynamic library diskLibPlugin.dll could hang after running for an extended period of time. If you encounter this issue, one workaround is to set the logging level to 4 or less (non-verbose) by setting transport.LogLevel in the configFile passed to VixDiskLib_InitEx(). Another workaround is to specify NBD explicitly in the TransportModes parameter of VixDiskLib_InitEx(). A fix has been identified and will be provided in the first update release.

  • vixMntapi on Windows can access only the first 2TB of a volume.

    The vixMntapi functions can be used to mount a Windows volume larger than 2TB. However, attempts to access files on the resulting mount may result in this error message: “The file system structure on the disk is corrupt and unusable. Please run the chkdsk utility on the volume.” Customers should not run chkdsk in this case. Even if the message does not appear, there is a hazard of corrupting the mounted volume if you attempt to change it, and a risk of getting incorrect data if the accessed file exists beyond the 2TB limit. There is no good workaround. Splitting the volume is not always possible or convenient.

  • SLES 11 GUI interferes with HotAdd proxy backup.

    When using SUSE Linux Enterprise Server (SLES) 11 SP2 as a backup proxy, the Gnome automount utility gvfs-mount tries to mount HotAdded virtual disks, causing I/O errors and forcible unmount after backup tries to access the disk device. The workarounds are to avoid running the GUI, or disable the gvfs-fuse daemon. You do not want Linux to mount the HotAdd virtual disks.

  • Changed Block Tracking could return full thin-provisioned disk.

    On very large thin-provisioned virtual disks (> 200GB) the first QueryChangedDiskAreas call could return the entire disk, rather than only the in-use areas of thin-provisioned disk. This results in excessively large backup data. It is expecially noticeable on Linux ext2 and ext3 file systems. A fix has been identified, and will be available in the VDDK 5.1 first update release. The workaround is to use the numeric changeId returned from the first snapshot, rather than the "*" (star) change ID, in the QueryChangedDiskAreas call.

  • Dependency on libexpat.so.0 XML parsing library.

    The VDDK libraries on a Linux proxy have a dependency on libexpat.so.0, but most current Linux distributions are shipping libexpat.so.1 instead. A workaround is to create a symbolic link to libexpat.so.1 with the name libexpat.so.0 in the /lib or /usr/lib directory.

The following known issues pertain to the VDDK 5.1 release but were reported previously for the VDDK 5.0 release.

  • Cannot load, unload, and reload VixDiskLib.dll in one process.

    A VDDK 5.x program cannot call VixDiskLib_Init() or VixDiskLib_InitEx(), then VixDiskLib_Exit(), followed by VixDiskLib_Init() or VixDiskLib_InitEx() again. Because the vixDiskLibPlugin must unload vmacore.dll on exit, the program crashes when Init or InitEx requests VixDiskLib.dll reload. Although the crash did not occur with VDDK 1.2.1 and before, programs should be revised to eliminate multiple Init or InitEx calls in a process.

  • Problem using UTF-16 characters in a pathname.

    VDDK supports only UTF-8 locales. When VixDiskLib_Open() is given a pathname containing UTF-16 characters, the virtual disk library fails to find the file. On Windows 2008 for example, the pathname Temp\vmware-système\*vm* contains è as a UTF-16 character, whereas VixDiskLib expects UTF-8. One workaround is to override the Temp pathname in the configuration file by setting the tmpDirectory key, using a non-UTF-16 pathname. For details, see documentation for VixDiskLib_InitEx(). Alternatively, software can be modified to use library routines such as iconv() for transcoding UTF-16 to UTF-8 and vice versa.

  • VDDK sample code may require additional header files.

    On some Linux distributions, you must include <stdio.h> and <string.h> near the top of vixDiskLibSample.cpp for the program to compile correctly.

  • At-sign (@) in datastore names.

    While all versions of vSphere allow the at-sign (@) in datastore names, the VixDiskLib API is unable to open VMDK files inside such datastores. You might want to modify your backup code to throw an error in such a case. The customer workaround is to avoid datastore names with the at-sign (@) when using VDDK.

  • Linux C++ libraries required.

    VDDK 5.0 and later require C++ library support, which many older Linux versions do not provide in the base install. When using an old Linux distribution as a backup proxy, customers might need to install C++ library packages from the distribution ISO.

  • VIX API and VDDK libraries are not compatible.

    If a process calls both VixDiskLib_Open() and VixHost_Connect() it results in a crash and possible core dump. Both VIX and VDDK libraries depend on the gvmomi library and call its init() function, which exits due to assertion failure. See KB 2016054 for a workaround.