--- title: vSAN Availability Technologies url: https://www.vmware.com/docs/vmw-vSAN-Availability-Technologies --- # vSAN Availability Technologies Resilience Capabilities for vSAN in VMware Cloud Foundation 9.1 October 12, 2026 ## Introduction Reliable and resilient access to applications and their data is a top priority for nearly all IT organizations. Virtualization afforded every environment to provide high levels of availability for virtual machines (VMs) and the data they serve. When using a classic three-tier architecture consisting of compute resources (vSphere hosts), networking, and storage, this meant that data availability was the responsibility of the storage solution. This typically was in the form of a traditional modular storage array using its proprietary hardware and dedicated storage fabric. The distributed storage architecture of vSAN changes this significantly. vSAN HCI clusters aggregate storage resources into the same hosts that provided the compute resources for the cluster. The hypervisor provided the distributed storage services exclusively through its software stack using commodity hardware. vSAN also offers a disaggregated approach to storage. This deployment option gives you the power of a fully distributed storage solution built into the hypervisor, but disaggregated away from the compute resources, so it can provide a centralized shared storage solution for one or many vSphere clusters powering VMware Cloud Foundation (VCF). ## Data Availability Concepts Maintaining availability of data means that a system must understand what types of failures may occur in a storage system. Some failures such as a storage device sending a failure code are recognizable. Other failures, such as a host outage, are not as easily recognizable because its absence is the only indicator of a known issue. Defining what a failure is can also be difficult, as a failure may not always occur in a clear and distinct way. For example, an interruption in a network link between hosts may be considered tolerable if for a brief period but deemed a failure if it exceeds a certain time threshold. When a failure is determined, a system must then be able to repair the condition or ameliorate it to the best of its ability at that moment. Any type of enterprise storage system must have all the proper intelligence and logic to handle these matters in a reliable and elegant manner. A distributed storage system like vSAN must factor in additional considerations due to its distributed architecture. vSAN factors in these complex considerations into a robust solution that maintains availability under a wide range of conditions. ## Scope of Topics This document provides details on how vSAN provides high levels of data availability. The content describes general behavior of vSAN. The information provided in this document will assume the use of vSAN 9.1, and/or VMware Cloud Foundation (VCF) 9.1. VCF deployments may have additional requirements and support limitations that fall outside of the scope of this document. For VCF environments, please refer to the Administration Guide for VMware Cloud Foundation for guidance as it relates to VCF. The release of vSAN in VCF 9.1 marks the 6th release of vSAN ESA, and as a result, this ## vSAN Availability Technologies vSAN Architectural Basics vSAN is a storage solution that provides shared storage for VMs running vSphere. It is a distributed storage solution, meaning that data is intelligently and automatically sprinkled across hosts that comprise the vSAN cluster to ensure data can remain available in the event of some known or unknown failure. It aggregates the storage devices across the hosts in a vSAN cluster and provides a single datastore that is easy to operate. This is quite different than shared storage using a traditional storage array, that provides resilience of storage through redundancy of discrete devices within the storage array, such as disks, storage controllers, and network uplinks. This distinction enables vSAN's distributed storage architecture to achieve greater scalability and resilience as data volumes and workloads increase. A standard vSAN cluster requires a minimum of 3 hosts, although we recommend 4 as a minimum in most cases. A single vSAN cluster can scale up to 64 hosts and be tailored to a size that best fits the needs of your organization. “vSAN Cluster Design – Large Clusters versus Small Clusters” details how you can use this flexibility to your advantage. Pair this capability with vCenter Server’s ability to manage many clusters within a VCF workload domain, and many workload domains that can be a part of a single VCF instance, and you have a platform that makes compute and storage resources extremely scalable. ## vSAN Objects and Components Unlike a clustered file system like VMFS used in a datastore residing on a storage array, vSAN uses a data structure that is analogous to an object store. It does this for several reasons, and are detailed in the post: “vSAN Objects and Components Revisited.” Objects are the entities that make up a given VM, such as virtual disks (VMDKs) and configurations. They are the smallest unit of management within vSAN and give administrators capabilities not found with traditional storage. These objects can be as large as 62TB in size. Components are smaller chunks of data that comprise an object. While components will be discussed in this document, components are simply an implementation detail of vSAN, and not a manageable entity. They are only discussed here and exposed in the UI because it can help an administrator better understand the concepts of object availability. They have their own limits in terms of the maximum size, and the maximum number of components that can reside on a host. The components of an object in vSAN ESA are a concatenation of two types: - **Performance leg components.** These are small components mirrored across 2 or more hosts and intended to temporarily store data and metadata in preparation to be written in a space efficient manner to the capacity leg. - **Capacity leg components.** These are the components distributed across the number of hosts necessary for the assigned data placement type (such as FTT= 2 using RAID-6). All these components play a role in determining availability of data during failure conditions through a vote count of components, as described later in this document. vSAN’s Cluster Level Object Manager (CLOM) automatically makes the placement decisions to satisfy the requirements of the assigned policy and the characteristics of the cluster. This is what makes the data resilient. The image below shows how a VM’s virtual disk object is placed across a series of hosts. It is assigned a storage policy of FTT= 2 using RAID-6 erasure coding. This results in a concatenation of 3 mirrored components in the performance leg that can tolerate up to 2 failures, and 6 components that make up a RAID-6 stripe with parity that can tolerate up to 2 failures. Components typically contain data payload and metadata of an object. In special circumstances, there may be a “witness” component. This is a very small component that simply contains metadata for the purpose of understanding the object availability. Witness components were common when using RAID-1 in vSAN OSA but are no longer used in ESA except for stretched clusters and 2-Node clusters, where the witness components reside on a witness host appliance. **Recommendation:** There is very little need to spend time trying to fully understand components and their relation to availability. ESA uses components in much more sophisticated ways compared to the OSA and can be quite confusing to decipher in complex failure scenarios. However, vSAN handles this logic so that you don’t have to. The information shared in this document is for informational purposes only. ## Accounting for Probabilities of Storage Failure vSAN maintains the availability of data by allowing the administrator to define resilience levels of the data that resides on a vSAN cluster. This is achieved using storage policies that are applied to data objects. The storage policy defines the outcome, or level of failures that the object can tolerate. vSAN only spreads object data across as many hosts as it needs to satisfy the resilience requirement. It does not spread data that comprises a single object across all hosts in the cluster. This is how vSAN decouples data availability considerations from cluster size. In contrast with other architectures, our object-based approach reduces risk of data unavailability as a cluster scales. It becomes less susceptible to the impacts of discrete failures as the cluster size increases. For more information on how vSAN’s erasure coding can be more resilient than what is found on traditional storage arrays, see the post: “Erasure Codes in VMware vSAN versus Storage Arrays.” ## Storage Policies and Component Placement Storage policies have historically played an integral part of vSAN data management, as they define the desired state, or outcome for one or more VMs. These outcomes fell into two categories: - Levels of data resilience - Discrete VM storage settings vSAN in VCF 9.1 introduces a new capability that automatically assigns and manages the optimal level of resilience of data stored on a vSAN cluster, based on the characteristics of the cluster. vSAN Auto-RAID is the successor to “Auto-Policy Management.” This feature allows the administrators to no longer worry about the optimal level of resilience. Storage policies still exist, but control VM storage settings like IOPS limits, and space reservations. For more information on Auto-RAID, see the post: “Auto-RAID in VMware vSAN for VCF 9.1 - Comprehensive System-Managed Data Resilience.” For versions prior to vSAN in VCF 9.1, setting data resilience was a manual setting. For example, one could set a higher level of resilience for their business-critical VMs and a lower level of resilience for their test/lab VMs. vSAN looks at the outcome defined in the storage policy (or Auto-RAID) and not only applies it to the VMs associated with it but will constantly look to ensure that it is compliant with the prescribed policy. And if not, it will make the appropriate changes automatically to regain the prescribed level of resilience. For example, a VM using a level of failure to tolerate (FTT) of 2 using RAID-6 erasure code will always stripe the data with parity across at least 6 hosts so that if two of the hosts holding that object fail, the data will remain available. vSAN has internal anti-affinity mechanisms in place so these components of an object will not reside on the same host. If there is a failure, vSAN will repair the data structure to regain its prescribed level of resilience, assuming it has sufficient hosts to do so. Topologies will dictate the data structure and placement of data, which ultimately determines the level of availability that topology can provide. As shown in the figure below, smaller clusters can store data using space-efficient erasure coding but will not be able to store the data at a higher level of resilience like a cluster with more hosts. Stretched clusters mirror the data across the geographic sites to ensure a site level resilience, while 2-Node clusters will provide host level resilience with a minimal amount of hardware. ### Data Placement in Standard Single Site vSAN Clusters The following shows how data is placed in a standard single site vSAN cluster. Note that the “Standard single site cluster using 1-2 hosts” is shown only to convey how data is placed when there is an insufficient number of hosts in the cluster. This is typically just a temporary condition for extremely small clusters (3 hosts) that have a host in maintenance mode or when initially building up a new cluster. ### Data Placement with Standard Single Site vSAN Clusters Using Fault Domains The following shows how data is placed when configuring a cluster using the vSAN Fault Domains feature. The data placement is very similar to a standard cluster but will sprinkle that data on any host within that defined fault domain (e.g., rack). For more information, see the post: “Using Fault Domains in vSAN ESA”. Note that Auto-RAID in vSAN for VCF 9.1 is fully compatible with clusters using the vSAN Fault Domains feature. The following shows how data is placed when configuring a cluster using vSAN stretched clusters. For more information, see the document: “vSAN Stretched Cluster Guide”. The following shows how data is placed when configuring a cluster using vSAN 2-Node clusters. For more information, see the document: “vSAN 2-Node Cluster Guide”. As noted below, secondary levels of resilience are currently not supported when using Auto-RAID in vSAN for VCF 9.1, but data is still fully resilient across the hosts. ### Data Resilience Set by Auto-RAID in vSAN for VCF 9.1 With Auto-RAID, you no longer need to worry about which resilience setting you should be using for your vSAN clusters. vSAN will automatically manage the resilience for data residing in a cluster, regardless of the size or type of vSAN cluster. Unlike previous efforts in the past to simplify resilience settings, vSAN Auto-RAID will only create a single policy within vCenter server, even though each one of your vSAN clusters may require different resilience settings. vSAN Auto-RAID in VCF 9.1 determines the optimal data placement for you. Below shows the outcome that it will select based on the cluster topology and conditions: **Standard single site clusters** - 6 or more hosts in cluster. Auto-RAID will use FTT= 2 using RAID-6. 1.5x object capacity overhead - 3-5 hosts in cluster. Auto-RAID will use FTT= 1 using RAID-5. 1.5x object capacity overhead - Fewer than 3 hosts in cluster. Auto-RAID will use FTT= 0. 1.0x object capacity overhead **vSAN cluster using Fault Domains feature** - 6 or more fault domains in cluster. Auto-RAID will use FTT= 2 using RAID-6. 1.5x object capacity overhead - 3-5 fault domains in cluster. Auto-RAID will use FTT= 1 using RAID-5. 1.5x object capacity overhead - Fewer than 3 fault domains in cluster. Auto-RAID will use FTT= 0. 1.0x object capacity overhead **Stretched clusters** - 6 or more hosts per site/fault domain. Auto-RAID will use a site disaster tolerance of a RAID-1 mirror, plus FTT= 2 using RAID-6. 3.0x object capacity overhead - 3-5 hosts per site/fault domain. Auto-RAID will use a site disaster tolerance of a RAID-1 mirror, plus FTT= 1 using RAID-5. 3.0x object capacity overhead - Fewer than 3 hosts per site/fault domain. Auto-RAID will use a site disaster tolerance of a RAID-1 mirror, plus FTT= 0. 2.0x object capacity overhead **2-Node clusters** - 6 or more storage devices per host. Auto-RAID will use a site disaster tolerance of a RAID-1 mirror, plus FTT= 0. 2.0x object capacity overhead. (secondary levels of resilience are not currently available for 2-Node clusters using Auto-RAID in 9.1) - 3-5 storage devices per host. Auto-RAID will use a site disaster tolerance of a RAID-1 mirror, plus FTT= 0. 2.0x object capacity overhead. (secondary levels of resilience are not currently available for 2-Node clusters using Auto-RAID in 9.1) - Fewer than 3 devices per host. Auto-RAID will use a site disaster tolerance of a RAID-1 mirror, plus FTT= 0. 2.0x object capacity overhead. ## Storage Policy Resilience Settings Note that due to the intelligence of Auto-RAID, it can easily handle the bootstrapping of a new cluster which may only have one host in it during the cluster creation process. Previously, one needed to use the “Force Provisioning” rule to accommodate for this temporary condition, but Auto-RAID will automatically handle all conditions where the “Force Provisioning” rule was needed. As noted above, in versions prior to vSAN for VCF 9.1, storage policies allow administrators to define the level of resilience desired for one or more objects. This is expressed as a storage policy rule by its level of failures to tolerate, or “FTT.” A failure in this context would relate to what vSAN defines as a fault domain. In a standard vSAN cluster, this would be a host. In a standard vSAN cluster using the “Fault Domains” feature, this could be a collection of hosts such as a room or a rack. In a stretched cluster, since the “Site disaster tolerance” already defines a mirror between sites, FTT would define the secondary level of resilience: hosts within a site. The FTT setting defines the level of resilience desired for the object. In a standard single site vSAN cluster, an FTT of 1, or FTT= 1 allows for one of the hosts holding that object to fail while remaining available. In that same type of cluster, an FTT= 2 allows for two of the hosts. Note that the FTT setting only applies to failures for the hosts that the object resides on, NOT the total number of failures within a cluster. This is why a vSAN cluster improves in its ability to maintain data availability as a cluster size grows, versus other distributed storage systems that tend to become more fragile as cluster sizes grow. As noted above, for stretched clusters and 2-Node clusters, the FTT level defined by a storage policy or Auto-RAID does NOT define the total resilience of a VM object, but the secondary level of resilience applied, if a secondary level of resilience is possible. For example, in a 12 host vSAN stretched cluster, where there are 6 data hosts in each site, you may see FTT= 2 assigned. This represents the number of hosts holding that respective object within each site that can fail while still maintaining availability. The “Site Disaster Tolerance” already defines that the data is mirrored across two sites, providing site level resilience. The “Failures to tolerate” storage policy rule also allowed for “No data redundancy.” Just as the name implies, this rule does not provide any ability to maintain availability if one of the resources is unavailable and is sometimes referred to as “FTT= 0.” Object data is striped without any parity, and in some parts of the UI (such as looking at data placement for an object) will be listed as “RAID-0.” For a completed cluster running production workloads, there were very few scenarios where FTT= 0 was recommended for use. With Auto-RAID in vSAN for VCF 9.1, the maximum level of resilience will automatically be applied. Auto-RAID may use an FTT= 0 using RAID-0 temporarily in conditions where resilience is not possible. Examples may be initially bootstrapping vSAN on a single host, or small 3-host clusters where a single host has been in maintenance mode for an extended period. In those scenarios, Auto-RAID will automatically apply resilience as the cluster as additional resources are added. While FTT= 3 is available for vSAN ESA (prior to Auto-RAID in VCF 9.1), it still uses data mirroring to provide redundancy. FTT=3 using mirroring has no practical use for ESA clusters, and should not be used. Since vSAN ESA provides RAID-6 erasure coding with no performance penalty, and fully supports durability components during planned and unplanned events, the additional capacity overhead of 3x (twice the amount of RAID-6 which is 1.5x) does not make it a practical option for most environments, and should be avoided. Previous versions of vSAN ESA attempted to make determining the optimal storage policy easier. “Auto-Policy Management” is the predecessor to vSAN Auto-RAID, and exists in ESA clusters running vSAN 8 U1 through 9.0. It created a cluster-specific storage policy with the optimal level of resilience for that cluster. It was an iterative step in an effort to simplify resilience settings, and is no longer needed for vSAN clusters running VCF 9.1. Use vSAN Auto-RAID for all vSAN clusters running 9.1 or newer. ## Storage Policy Data Placement Scheme While the storage policy rule “outcome” of FTT= 1 or FTT= 2 establishes the level of resilience desired, the description, the data placement scheme defines how it is achieved. For example, prior to Auto-RAID in vSAN for VCF 9.1, the “Failures to tolerate” rule may give options such as: - 1 failure – RAID-1 (Mirroring) - 1 failure – RAID-5 (Erasure Coding) - 2 Failures – RAID-1 Mirroring - 2 Failures – RAID-6 (Erasure Coding) A RAID-1 mirror means that vSAN simply keeps a synchronous copy of the object data somewhere else on the cluster to provide the prescribed level of resilience, such as FTT= 1. RAID-5 or RAID-6 indicates the use of an erasure code, which is the ability for vSAN to store data in components in an intelligent stripe with parity across a series of hosts to provide the prescribed level of resilience. Erasure codes are much more space efficient than simple mirroring but may require more hosts to achieve the desired level of resilience. Note that the methods noted above are obfuscated by Auto-RAID, as it will automatically use the optimal data placement scheme based on the conditions of the cluster. ## vSAN Availability Technologies vSAN ESA can store data using RAID -5/6 erasure coding without any performance penalty. Thus, RAID-1 mirroring should not be used in anything other than site-level resilience for stretched clusters, and host-level resilience in 2-Node clusters. With Auto-RAID in vSAN for VCF 9.1, a RAID-1 mirror will never be used for anything other than site-level resilience for stretched clusters, and host-level resilience for 2-Node clusters. When using RAID-5/6 erasure coding, each component in the performance leg contains a combination of data and parity data. There is no dedicated parity component. There is also not any dedicated witness component. Each data placement scheme has a minimum number of hosts required in order store the data in a non-failure state. For example, FTT=1 using RAID-5 erasure coding requires just three hosts, while FTT=2 using RAID-6 erasure coding requires a minimum of 6 hosts. In a standard vSAN ESA cluster, the recommended minimum number of hosts when using FTT=1 using RAID-5 is 4 hosts. The recommended minimum number of hosts when using FTT=2 using RAID-6 is 7 hosts. A cluster with an additional host beyond the required minimum allows clusters to regain their prescribed levels of resilience in the event of a sustained failure. It also helps improve data durability, where newly written data during failure conditions through the use of special “durability components” in vSAN. See the section “Availability Handling” for more information. In vSAN 8 through vSAN 9.0, ESA had an ability to use an adaptive RAID-5 erasure coding scheme for certain cluster sizes. In vSAN for VCF 9.1, when Auto-RAID does not have sufficient hosts to use FTT=2 using RAID-6 (4+2), it will using FTT=1 using RAID-5 (2+1). Recommendation: If you are running on vSAN 8 U1 to vSAN 9.0, use Auto-Policy Management. For vSAN in VCF 9.1, use vSAN Auto-RAID. New clusters in VCF 9.1 will always use vSAN Auto-RAID. Existing clusters upgraded will keep their manually or semi-automatic storage policies. Simply change the storage policy of all objects to “vSAN ESA Auto-RAID Policy” or enable the “Apply Auto-RAID to all objects toggle” for the cluster. For clusters running version 9.0 or older, an FTT=2 can be achieved through RAID-1 mirroring. This policy rule is not recommended for vSAN ESA clusters, and will not be used when Auto-RAID establishes the optimal level of resilience for a cluster. This “2 Failures – RAID-1 Mirroring” setting stems from the early days of vSAN OSA, where one wanted to achieve higher resilience without the performance penalty of RAID-6 erasure coding. While it could be achieved on as few as 5 hosts, it resulted in very poor space efficiency, requiring 3x the capacity to ensure the prescribed resilience. For clusters consisting of 5 hosts or less, the automated setting of FTT=1 using RAID-5 erasure coding is the optimal data placement scheme offering high levels of durability that wasn’t possible with early versions of vSAN. This enhanced durability comes from two capabilities not in older versions of vSAN: - The use of durability components. These are special components created when vSAN has detected that some components are not available (due to planned or unplanned events), and will ensure that subsequent writes are always written at the highest level of resilience. For more information, see the details on durability components described later in this document. - A smaller erasure code results in more resynchronization targets. vSAN ESA (especially in 9.1) will always use a 2+1 erasure code in clusters smaller than 6 hosts. This means that data can be reconstructed quickly to two other hosts to regain their prescribed level of compliance in the event of a failure. - Smaller boundaries of failure. vSAN OSA needed to factor in the potential failure of disk groups within a host. vSAN ESA no longer has this concept and can minimize the amount of data to be resynchronized if discrete devices fail. For more information, see the post: “The Impact of a Storage Device Failure in vSAN ESA versus OSA.” ## Fault Domains vSAN uses a construct known as a “fault domain” to help it distribute data in a resilient way. By default, vSAN treats each host as a fault domain, as shown in the image below. In a standard, single site vSAN cluster, this helps keep data available in the event of a discrete host failure. This “Fault Domains” feature can be used to define a group of hosts to create a logical boundary of failure. It can use this information to write the data in such a way that if that defined fault domain is offline, the data will remain available. The manual creation of fault domains is often used to create groups of hosts in a cluster that represents their placement in a rack. This helps create rack-level resilience for a single site vSAN cluster. For more guidance on the use of vSAN Fault Domains, see the post: “Design and Operation Considerations when using vSAN Fault Domains.” The post “Using Fault Domains in vSAN ESA” will provide additional guidance as it relates to vSAN ESA. vSAN stretched clusters use a form of this construct, where the two data sites and the witness site are each defined as a fault domain. 2-Node vSAN clusters also use this method where the two data hosts and the witness site are each defined as a fault domain. The establishment of fault domains is automatically created for you when you select either one of those deployment types. vSAN in VCF 9.1 provides additional functionality for site level resilience through the use of a new “site maintenance mode” feature, as well as a “manual site takeover” capability. See the “vSAN Stretched Cluster Guide” for more information. Unlike a traditional vSAN cluster, stretched clusters and 2-Node clusters allow you to establish a secondary level of resilience. (For vSAN in VCF 9.1, secondary levels of resilience are temporarily limited when using Auto-RAID in 2-Node clusters). In the case of a stretched cluster, this can help maintain data availability not only after a full site failure but additional failures within the remaining site. For more information including an exhaustive list of failure handling scenarios with vSAN stretched clusters, see the “vSAN Stretched Cluster Guide.” Recommendation: Build a cluster with at least one more host than the minimum required by your storage policy. Upon a sustained outage or maintenance of a host, this will allow vSAN to automatically repair the data impacted by the failed host and reestablish the prescribed level of resilience assigned by the storage policy. If there is not an available host to rebuild. ## vSAN Storage Clusters vSAN can be deployed as an aggregated vSAN cluster, where compute and storage resources are provided by the same cluster. It can also be deployed as a vSAN storage cluster (previously known as “vSAN Max”). Much like a traditional storage array, a vSAN storage cluster can serve as a centralized shared storage solution for vSphere clusters. For more information, see the post: “vSAN Max and the Advantage of Scalability.” On the topic of availability, many of the same concepts and considerations for vSAN HCI clusters apply to vSAN storage clusters. See the “Disaggregated Storage using vSAN Storage Clusters” section of the vSAN FAQs, and the “vSAN Storage Clusters Design and Operational Guidance” document for guidance and recommendations. ## Application versus Infrastructure-Level Availability vSAN provides availability of data and systems at the infrastructure level. Sometimes, applications can provide their own high levels of availability through application-level replication. How do they compare? See the post: “Application Versus Infrastructure-Level High Availability with vSAN in VMware Cloud Foundation” for a better understanding of how one approach compares to the other. ## Determining Availability of Data Components of an object help determine the availability of an object. This is achieved through a vote count of components. This vote count of components helps determine if there is enough data to deem the object as available. This threshold is known as a “quorum” and is a safety mechanism to help vSAN determine under failure conditions when an object should be available or when it should be marked as unavailable to preserve its integrity. An object must have more than 50% of its components accessible to meet quorum and deem the object as available. While most of the time each component has a value of “1” in vSAN’s quorum voting mechanism, it may assign a value of two for some objects that have an even number of components. This is an implementation detail, and generally it is not necessary to know. vSAN stretched clusters and vSAN 2-Node clusters have a unique ability to change the vote count of components in very specific situations, where a data site is offline due to a planned or unplanned event, followed by the witness site being offline for a planned or unplanned event. Assuming sufficient time between outages, vSAN will recalculate the votes to ensure that the one remaining site will remain available, even though a data site and a witness site are unavailable. For more information, see the “Adaptive Quorum Control” topic in the vSAN Stretched Cluster Guide. vSAN is continuously monitoring the availability of all data and depending on events in a cluster (such as a transient network error, or a host failure), will determine how to proceed with regaining its prescribed level of resilience. ## Component States vSAN components will typically show one of the following states described in more detail below: ### Active The most common state is “Active” which means the component is accessible and up to date. ### Reconfiguring This state is usually found when a change to a storage policy is made, or a new storage policy is assigned to an object. ### Absent This state represents components that can no longer be accessed. It does not define why it can’t be accessed, but rather that they are simply not available. This is like an all paths down “APD” event with traditional storage. For example, in an 8-host cluster, imagine an object using a storage policy of FTT= 2 using RAID-6. The data with parity would be spread across 6 of the hosts. If power was lost on one of the hosts storing a portion of that object, that specific component would be marked as “absent.” Since vSAN is unaware of the reason for its absence, it will wait for a period for the component to be marked as active. This helps accommodate transient outages such as a temporary network error, or a host restart. Once it exceeds this time (the default is 60 minutes), it will begin to rebuild the component somewhere else in the cluster, assuming another previously unused fault domain (host) is available. ### Degraded A degraded component state is typically the result of the system receiving SCSI sense codes that indicate a device failure. This type of permanent device loss (PDL) will initiate rebuild or repair activities immediately since the failure is known. Note that pulling a device to test availability would only simulate an APD event described above, and not a PDL described here, since the system never received a sense code. ### Stale Stale components, or “Active – Stale” as shown in the UI, represent components that are available but are outdated. vSAN uses sequence numbers to verify a component has the latest updates. This sequence number is normally kept consistent across components that make up an object. When a change to the object occurs, the data is written to both replicas, for example, and the sequence number is updated for the components. If a component is active, but its sequence number is different/older than the current sequence number for the object, the component is marked as Active - Stale. This usually occurs when the components of an object go offline and come back online concurrently at different times. Those stale components have an older sequence number and represent that they may not reflect the latest updated data. To preserve data integrity, vSAN marks these as “Active – Stale” until the components can be updated with the same sequence number. ## vSAN Availability Technologies Recommendation: Avoid changing the default rebuild timer setting of 60 minutes. A 1 hour timeout value provides a good balance of avoiding unnecessary rebuilds while minimizing the risk of downtime. If a situation occurs where a timelier rebuild is desired, it is possible to trigger a rebuild using the Repair Objects Immediately in the vSAN Health Check user interface. Attempting to take a snapshot of a VM in vSAN OSA will fail if the storage policy for that VM is not fully compliant. This is not an issue in vSAN ESA. ## Availability Handling The storage of data across hosts in a resilient way is what allows vSAN to maintain availability under a wide range of conditions. The event handling below shows outcomes of: - Data available and resilient to its prescribed level of resilience - Data available and degraded to a lesser level of resilience - Data available but with no level of resilience - Data unavailable Unavailability of data does not mean “data loss.” It simply refers to the state of availability for the VM to use or access the data. vSAN’s data path always commits data to persistent media using advanced consensus algorithms to ensure data consistency. It does this prior to ever responding to the guest VMs with a write acknowledgement. When discussing scenarios of object availability in vSAN and the resilience of that object per its prescribed storage policy, a failure only applies to the hosts that the object resides on, NOT the total number of failures within a cluster. An object, such as a VMDK is a relatively small boundary of data. vSAN will only place this on a sufficient number of hosts to comply with the storage policy resilience setting. Therefore, other failures that occur within a cluster will have no impact on the object if it does not use the other hosts that failed. As recommended in the “Storage Policy Data Placement Schemes” section of this document, we typically advise your vSAN cluster consisting of at least one more host than the minimum needed for the desired storage policy. This allows for object data to automatically regain their prescribed level of resilience in the event of a sustained host outage. The following scenarios are going to demonstrate the availability handling of failures in the following conditions: - Availability handling for an object assigned FTT=2 using RAID-6 in a 7 host cluster. This will step through a sequence of failures to help describe when and how the data regains its prescribed level of resilience after a failure and then remains available after additional failures. It will then show the threshold in which it is no longer available. - Availability handling for an object assigned FTT=1 using RAID-1 or RAID-5 in a 3 host cluster. This will step through a sequence of failures on a cluster using an absolute minimum number of hosts for a standard vSAN cluster. For clarity, the illustrations may simplify the component quantity and placement. The illustrations also do not include “durability” components, which simply improve the durability of the most recently written data in times of multiple failures. ## Availability Handling for an Object Assigned FTT=2 using RAID-6 in 7 Host Cluster This group of scenarios will describe the handling of an object on a vSAN ESA cluster with a storage policy that has FTT=2 using RAID-6. A RAID-6 object will have its capacity leg components spread across 6 hosts, and its performance leg components spread across 3 hosts. FTT=2 using RAID-6 will be the most common storage policy in a standard, single site vSAN ESA cluster with 6 or more hosts. Note that vSAN Auto-RAID will automatically ensure the maximum level of resilience based on the characteristics of the cluster. ### Example 1: One Host Offline in a 7 Host Cluster In this first example, a host goes offline. The object remains available, but in a degraded state with a reduced effective resilience of FTT=1. After a 60 minute delay, that portion of the stripe with parity will be rebuilt on the host that does not contain the object. It will then regain its full prescribed level of resilience of FTT=2. ### Example 2: Two Hosts Offline in a 7 Host Cluster In this second example, an additional host goes offline. The object remains available, but in a degraded state with a reduced effective resilience of FTT=1. The object cannot regain its prescribed level of resilience because there are not any hosts (fault domains) as a rebuild target. ### Example 3: Three Hosts Offline in a 7 Host Cluster In this third example, yet another host goes offline. The object remains available, but in a degraded state with no resilience - equivalent to an FTT=0. The object cannot regain its prescribed level of resilience because there are not any hosts (fault domains) as a rebuild target. It cannot remain any more host failures beyond what is shown. ### Example 4: Four Hosts Offline in a 7 Host Cluster In this fourth example, yet another host goes offline. The object is no longer available because a RAID-6 erasure code creates a stripe with double-parity across 6 hosts. It can tolerate 2 failures, but must have 4 of the 6 hosts available; otherwise, the object will be unavailable. ## Availability Handling for an Object Assigned FTT=1 using RAID-5 in 3 Host Cluster This scenario describes the failure handling of a standard vSAN cluster using the minimum of 3 hosts. This will demonstrate the behavior of vSAN in this specific condition. The general principles shown in the illustrations below apply to objects in a 3 host cluster using RAID-5(2+1). Note that component counts and placement for RAID-1 will be slightly different than what is shown in the illustration, but the behavior will be similar. ### Example 1: One Host Offline in a 3 Host Cluster In this first example, a host goes offline. The object remains available, but in a degraded state with a reduced effective resilience of FTT=0. The object cannot automatically regain its prescribed level of resilience because there are not any hosts (fault domains) as a rebuild target. If the host comes back online, vSAN will be able to resynchronize the data to regain the prescribed level of resilience. ### Example 2: Two Hosts Offline in a 3 Host Cluster ## Durability of Data to Account for Multiple Events Under times of planned or unplanned events, such as a host maintenance mode event or host failure, vSAN may create what are known as “durability components.” These are special components that contain only updated blocks at the time a planned or unplanned outage occurred. For example, as illustrated in the figure below, imagine a VM object in a 4 host vSAN ESA cluster using a storage policy assigned with FTT= 1 using RAID-5. In a normal operating state, the data with parity is striped across three hosts. Let’s imagine the third host from the left is placed into maintenance mode using “Ensure accessibility.” The object will remain available but no longer be resilient. At this time, vSAN creates a temporary durability component to capture all of the latest writes, in addition to the writes captured in the remaining stripe. If another host fails at this time, the object would no longer be available, but if the host originally placed into maintenance mode was brought back online, the delta changes in the durability component would be merged with the stripe, and the object would be available again. ## The Role of the Network with vSAN Since vSAN is a distributed storage system, it relies on the network to write the data in a resilient way and read the data in a reliable manner. This includes I/O from guest VMs, as well as “back-end” I/O activity that occurs to rebalance and repair data. Any transient or permanent interruptions in the network will impede vSAN’s ability to write or read the data requested. The network that is used for vSAN is just as important as a dedicated storage fabric used with a traditional storage array. ### Transport Method vSAN uses TCP (unicast traffic) over Ethernet to transmit I/O payload across the hosts that comprise the cluster. On each host within a vSAN cluster, a VMkernel port is tagged for vSAN traffic. This VMkernel port is used to send traffic using vSAN’s lightweight, purpose-built Reliable Datagram Transport (RDT) protocol. This protocol is used to help vSAN quickly build up and tear down sessions, and a myriad of other responsibilities. vSAN can optionally use RDMA over Converged Ethernet (RoCE v2). While TCP is highly durable, and can reliably deliver network traffic under poor conditions, RDMA can be ideal for high performance, low latency environments. RDMA does require special configurations on the switchgear and hosts and will also have other limitations. ### Network Redundancy Redundancy of a network ensures that any failure of a Network Interface Card (NIC) port, or a switch will not interfere with the connectivity to the rest of the cluster. For vSAN clusters, this redundant network communication is achieved in a very similar manner to that of traditional vSphere clusters. - Hosts with two NICs consisting of one or more ports for each NIC. - Hosts configured with a teaming policy for that vSAN VMkernel port that will allow it to fail over to another port in the event of a failure. - A port from each NIC on the host connecting to one of two switches. The traits listed above would characterize a vSAN cluster that could continue to reliably transmit storage traffic in the event of communication failure due to discrete failures of a NIC and/or a switch failure. **Recommendation** For vSAN traffic, use a NIC teaming policy of “Active/Standby” with “Route Based on Originating Virtual Port ID.” This will offer the most deterministic path for vSAN traffic, which is critical for storage performance and consistency. Note that VCF will by default use a teaming policy of “Active/Active” with “Route Based on Physical Load” otherwise known as “Load-Based Teaming.” While this approach teaming policy is adequate for VM port groups, it does not offer a deterministic path for storage traffic. You can easily override the VMkernel port settings for just the VMkernel port tagged for vSAN so that it is using “Active/Standby.” Overriding this vSAN VMkernel traffic to “Active/Standby” IS supported in VCF. ### Network Partitioning In a distributed storage system like vSAN, a certain type of failure condition must be accounted for. All hosts in a vSAN cluster must be able to communicate with each other. But there may be a condition where one subset of hosts in a cluster can communicate to each other, but not to the other hosts in a vSAN cluster. This is known as a “network partition” and must be accounted for in the proper way. An element of vSAN known as the Cluster Monitoring, Membership and Directory Services (CMMDS) is responsible for monitoring communication between the hosts in a vSAN cluster. When communication between hosts is sufficient, vSAN will indicate (in the UI, APIs, etc.) that there is a single partition group, reflecting a desired condition. ## Integration with vSphere High Availability (HA) vSphere HA uses the same network connections as vSAN when vSAN is enabled. This helps ensure that vSphere HA and vSAN see the same network partition(s) in a cluster. Having a consistent view of network partitions across vSAN and HA enable accurate and reliable responses to host isolation and multiple network partitions. vSphere HA is inherently reactive, meaning that it will only restart VMs if they are unavailable. vSAN also supports the use of vSphere Proactive HA in conjunction with a plug-in provided by the server OEM. This optional feature is capable of proactively removing VMs off of hosts suspected of imminent failure and may be helpful in maintaining the highest levels of uptime in your vSAN environment. ## Resynchronization Activity vSAN will ensure that data is stored per the desired resilience outcome defined in the object’s storage policy. It achieves this desired outcome through “resynchronization” activity. This is I/O processing that is initiated by the system to ensure the data complies with the defined storage policy. Resynchronization may occur for several reasons, including but not limited to: - Entering a host into maintenance mode. - Rebalancing data across the cluster - Storage policy changes - Regaining prescribed level of resilience after a failure Resync activity can be easily viewed in the UI, as shown below. ### Resyncing Objects view The guest VM is unaware of this resynchronization activity since it was not issued by the guest VM. There can be times where a large number of resynchronizations may have some impact on guest VM performance, but this was typically associated with earlier versions of vSAN, and is not a common issue anymore, especially with vSAN ESA. ### Resynchronization Activity from Failures As noted in the last bullet point above, resynchronization activity may occur after some type of failure has been recognized. For example, in an 8-host cluster, imagine an object using a storage policy of FTT= 2 using RAID-6. The data with parity would be spread across 6 of the hosts. If power was lost on one of the hosts storing a portion of that object, that specific component would be marked as “absent.” Since vSAN is unaware of the reason for its absence, it will wait for a period for the component to be marked as active. This helps accommodate transient outages such as a temporary network error, or a host restart. Once it exceeds this time (the default is 60 minutes), it will begin to rebuild the component somewhere else in the cluster, assuming another previously unused fault domain (host) is available. How it rebuilds the data will depend a bit on the data placement scheme used. A repair of an object using erasure coding may involve recalculating the data and parity bits to reconstruct the remainder of the stripe. A repair of an object using mirroring may simply copy the data from the other replica. vSAN may also use the special durability components for repair processes, which can speed up repairs. ## Minimizing Resynchronization Activity Recent versions of vSAN are dramatically better at handling resynchronizations. These improvements come from a reduced boundary of failure, better processes to minimize the amount of data to resynchronize, and intelligent throttling to minimize the impact on VMs during resynchronization. - Reduced boundary of failure after a storage device failure. (ESA) - Much faster processing of resynchronization data. (ESA) - Adaptive Network Traffic Shaping. (ESA) - Adaptive Resync. (ESA and OSA) - The use of Durability Components for faster repairs. (ESA and OSA) - Smart Efficient repairs choose the best option to rebuild based on the conditions. (e.g. rebuild new replica versus resync existing components). (ESA and OSA) ## Speeding up Resynchronization Activity Resynchronization are an automated mechanism that does not require any intervention from the user. If you are interested in designing an environment to ensure the fastest level of recovery in the event of perhaps a host failure, two factors will impact this recovery time. - **Network speed.** This is where faster networking like 100Gb networks can deliver much faster recovery rates than 25Gb networking. - **Cluster host count.** When reconstructing the data from a failed host, clusters with larger host counts will be able to regain the prescribed level of resilience faster because there are more hosts to rebuild the data. ## Maintenance Mode Entering a host into maintenance mode when configured as a vSAN cluster is a bit different than a traditional vSphere cluster. Since a vSAN cluster aggregates storage devices in the hosts that comprise the cluster, placing a host into maintenance mode impacts the overall storage capacity. When you place a host into maintenance mode, you may impact the level of failures that some objects can tolerate. At the time that a host is entered into maintenance mode (EMM), a pre-check simulation is performed on the data that resides on the host so that vSAN can communicate to the user the type of impact the EMM will have, all without moving any data. If the pre-check results show that a host can be seamlessly placed in maintenance mode, decide on the type of data migration. Consider the storage policies that have been applied within the cluster. Some migration options might result in a reduced level of availability for some objects. The EMM option chosen will dictate the behavior and availability of data on a vSAN cluster. ### Host maintenance mode options in vSAN - **Full data migration** — Evacuate all components to other hosts in the cluster. This option maintains compliance with the FTT number but requires more time as all data is migrated from the host going into maintenance mode. It usually takes longer for a host to enter maintenance mode with Full data migration versus Ensure accessibility. Though this option assures the absolute availability of the objects within the cluster, it causes a heavy load of data transfer. This might cause additional latency if the environment is already busy. When it is recommended to use Full data migration: - If maintenance is going to take longer than the rebuild timer value. - If the host is going to be permanently decommissioned. - If you want to maintain the FTT method during the maintenance. - **Ensure accessibility (default option)** — Instructs vSAN to migrate just enough data to ensure every object is accessible after the host goes into maintenance mode. vSAN searches only for data with RAID-0 and move/regenerate them on a host different than the one entering in maintenance mode. All the other objects with RAID-1 and higher should already have at least one copy residing on a different host within the cluster. Once the host comes back to operational, the data components left on the host in maintenance mode update with changes that have been applied on the components from the hosts that have been available. Keep in mind the level of availability might be reduced for objects that have components on the host in maintenance mode. This maintenance mode is intended to be used for software upgrades or node reboots. Ensure accessibility gives the opportunity to avoid needless Full data migration since the host will be back to operational in a short time frame. It is the most versatile of all EMM options. The “Ensure accessibility” option will behave differently than in object creation or failure handling conditions. It puts availability of that data at the highest level of importance, with no regard to the chosen data structure or anti-affinity rules assigned by the storage policy. It will essentially defy any and all elements of the storage policy and how the object is built, which typically dictates availability, for the single purpose of maintaining availability of the object data. - **No data migration** — No data is migrated when this option is selected. A host will typically enter maintenance mode quickly with this option, but there is a risk if any of the objects have a storage policy assigned with FTT= 0. The “No data migration” option is best suited for full cluster shutdowns, or environments where network changes are being made. ### Recommendation In most cases, using the “Ensure Accessibility” option will be your best choice when placing a host into maintenance mode. This will minimize the amount of data movement and will make your server maintenance operations much faster and more efficient. For more guidance on the use of maintenance mode in a vSAN cluster, see the “vSAN Operations Guide.” ## Handling Failed Storage Devices Storage devices are one of the more challenging entities to determine failure, as their operational state is not always a binary “working” versus “failed” matter. In many cases, a device may become highly unresponsive or perform poorly but still not be in a working state. As vSAN and the hardware available on the market have improved, several enhancements have been made to determine the operational state of storage devices. This, paired with the reduced impact of a storage device failure in vSAN ESA makes device handling much more sophisticated, and less impactful than in previous editions. ### Degraded Device Handling (DDH) Degraded Device Handling (DDH) is a monitoring technique within vSAN that will detect high device latency thresholds and patterns to these thresholds that suggest imminent failure of storage devices in the capacity tier. This symptom-based approach attempts to detect impending failures of storage devices. It was originally developed to help work around the shortcomings of the industry standard Self-Monitoring, Analysis, and Reporting Technology (S.M.A.R.T. or SMART). This standard never materialized as a reliable or consistent way to poll device telemetry data from SAS and SATA storage devices. Iterative improvements have been made to DDH across multiple releases. Much of the effort went to reducing false positives by using a different algorithm to detect high latency. For example, more advanced logic will observe performance traits of a time period by breaking a configurable time period (4-hour default) into a series of configurable intervals (20-minute default). A device is considered unhealthy if performance traits exceed the configured average performance trait for that device. These are randomly selected non-contiguous intervals to better represent a pattern of degrading performance. If an issue is detected with a storage device, alerts will be triggered with those components being marked as absent. vSAN will strive to ## Degraded Device Handling DDH is a symptom-based solution and can be prone to false positives. For example, high latency can be a leading indicator of a failing device, but it can also be an indicator of high sequential writes using large I/O sizes or a period in which garbage collection activities occur. Therefore, DDH should not be viewed as a definitive technology to detect failing devices, but a complementary mechanism that provides additional insight and protection for storage devices behaving erratically. ### Recommendation If you have experienced DDH triggering devices as failed, consider moving to vSAN ESA. vSAN ESA eliminates the use of dedicated storage controllers on hosts, as well as legacy storage devices using SAS and SATA protocols. These were much more prone to non-deterministic levels of latency. DDH has been improved for vSAN in VCF 9.0. For vSAN OSA, DDH has been improved to properly account for behaviors found in the caching devices of a disk group. For vSAN ESA, DDH has been adjusted to better account for ESA’s storage stack to more accurately identify symptoms of devices that may suggest impending device failure. ## NVMe Device Endurance Tracking (ESA only) vSAN ESA can take advantage of a new SMART standard specification exclusively for NVMe-based storage devices. This allows vSAN to check for and alert against any NVMe storage device that is nearing its expected number of device writes. These alerts can be easily customized within vCenter Server. Unlike previous specifications, SMART data polling has little to no effect on the storage device itself, which ensures this as a reliable way to move forward on intelligent device telemetry. For more information, see the post: “Health and Performance Monitoring Enhancements in vSAN 8 U2” and “Enhanced Intelligence in vSAN 8 U3 and VMware Cloud Foundation 5.2.” ## Low-Level Metadata Resilience (ESA only) vSAN ESA introduced a capability that helps address Unrecoverable Read Errors (UREs) in storage devices. UREs typically occur throughout the lifespan of a storage device. While vSAN employs all different types of resilience techniques for data and metadata in vSAN, we wanted to make the ESA more robust and efficient in the event of these UREs. ESA will mirror some low-level metadata on each discrete device to help ensure that the metadata created and managed by vSAN’s local log structured object manager (LSOM) is resilient on the device in the event of a discrete, 4KB URE. Even though the mirrored metadata resides on the same device, given the nature of UREs and their typical impact radius (typically at a 4KB granularity), this approach provides a statistically reasonable level of resilience to protect against UREs. Best of all, this capability is completely transparent to the administrator and requires no interaction or management. ## Proactive Hardware Management (ESA only) vSAN 8 U3 (and VCF 5.2) introduced a new framework that provides a uniform, vendor-neutral way to take device metadata which can serve as leading indicators of impending failure to then predict and present its results in the form of health findings in Skyline Health so that appropriate action can be taken. This uses the vLCM framework already in place, so ReadyNode vendors can adopt it easily. For devices that have been proactively detected to fail soon, RMAs for these devices are guaranteed by the server OEM. It will surely prove to be a powerful way to expose key device telemetry details to address health-related issues. For more information, see the post: “Enhanced Intelligence in vSAN 8 U3 and VMware Cloud Foundation 5.2.” ## Availability versus Protection Availability, resilience, durability, protection, and disaster recovery are often conflated in ways that can be challenging for users to clearly understand the differences. The information below will shed some light on their similarities and differences as they relate to vSAN and its availability technologies. ### Availability of Data Storage systems typically focus on the availability of data as it relates to the systems that are using that data. Availability is closely tied with “resilience” as this is the typical way that high availability is expressed and implemented in systems. vSAN has a plethora of techniques mentioned throughout this document to ensure that data remains resilient to failure or highly available. This can include anything from vSAN’s data placement schemes to stretched cluster topologies that maintain the availability requirements for customers. vSAN even has durability mechanisms in place (such as durability components) that help data to remain intact and accessible over multiple disruptions. ### Protection of Data Protection of data typically refers to the steps taken for a more definitive step, such as a recovery to a specific date and time. This is achieved through other mechanisms such as asynchronous replication like vSphere Replication or backups using third-party tools that use special VMware APIs for Data Protection (VADP). Protection strategies are commonly built around a “3-2-1” rule. This refers to the notion of having three copies of data, with two backups using different media types. ### Additional Resources The following are a collection of useful links that relate to bandwidth sizing for vSAN stretched clusters. - vSAN Interactive Infographic. This tool allows you to dynamically choose various configurations and failure scenarios to better understand how vSAN maintains supreme levels of availability. - vSAN Failure Handling Guide. Provides information on how vSAN handles specific failure scenarios. - Performance Recommendations for vSAN ESA. This is a collection of recommendations to help achieve the highest levels of performance in a vSAN ESA cluster. Many of these same recommendations apply to vSAN storage clusters. - vSAN Proof of Concept (PoC) Performance Testing. This is a collection of recommendations that will guide users to test the performance of a vSAN cluster. While it is currently written for the OSA, many of the testing methods used are also applicable to the ESA. - Design and Sizing for vSAN ESA clusters. This post offers some nice guidance on using the vSAN Sizer for the ESA that summarizes some key points that can be found in the VMware vSAN Design Guide. - vSAN Network Design Guide. This network design guide applies to environments running vSAN 8 and later. - vSAN technical blogs. Stay up to date on the most recently published technical information about vSAN. These posts are created by the vSAN Technical Marketing team. - VMware Resource Center. The location for design guides, operations guides and other technical white papers on vSAN. These assets are created by the vSAN Technical Marketing and Product Enablement teams. - Official vSAN documentation. The location for all “how to” documentation on vSAN. ### About the Author Pete Koehler is a Product Marketing Engineer in the VCF division at Broadcom. With a primary focus on vSAN, Pete covers topics such as design and sizing, operations, performance, troubleshooting, and integration with other products and platforms.