vSAN Stretched Clusters & VM Swap Files

My buddy Pete Koehler put together a good post a few days ago about SPBM Policies with vSAN Stretched  Clusters: vSAN Operations: Use separate SPBM policies for VMs in stretched clusters

In that post, he covers definition changes from Number of Failures to Tolerate (FTT) which was available in pre vSAN 6.6 builds, to Primary Failures to Tolerate (PFTT) in vSAN 6.6 and higher. This change was added to address the addition of local protection within a Stretched Cluster. This is accomplished using the secondary rule, Secondary Failures to Tolerate (SFTT), to determine the local protection within a Stretched Cluster Fault Domain (Preferred or Secondary).  Pete covers this in the above post.

A question came up last week specific to the behavior of a VM’s swapfile in vSAN when using vSAN Stretched Clusters. There has been some confusion with my documentation on StorageHub, which I’m currently updating, but I wanted to post something specific to how a VM’s swapfile behaves.

I’ve setup a Stretched Cluster with a single VM on it for a little clarity. Once site is “Denver” and one is “Colorado Springs” in my demo environment. Since I’ve moved to CO, I figured I’d use some naming that has local flair.

Pre vSAN 6.6 Stretched Cluster Behavior (vSAN 6.1, 6.2, & 6.5)

In vSAN Stretched Clusters before 6.6, each object had 1 component in each Fault Domain (Preferred/Secondary) and a Witness component on the vSAN Witness Host. *If an object is greater than 255GB in size, it will be broken into multiple components (on each site) for every increment of 255GB.

Here’s a standard policy for Pre-vSAN 6.6 Stretched Clusters:
PFTT=1 (Protecting across sites)
Failure Tolerance Method = Mirroring (one replica component in each site)
Force Provisioning = No (requires both sites to be up) or Yes (can still provision with one site offline)

Here’s what the placement of a VM Home looks like:One component is in the Preferred site, one is in the Secondary site, and then there is a Witness component on the vSAN Witness Host.

Looking at the vmdk, the same can be seen (the vmdk is <255GB)
Looking at the Swap file, it is also the same. One replica in each site, and a Witness component on the Witness host.

Notice though that the VM SWAP object does not have a policy associated with it.Hard coded policy for VM SWAP object
I’ve covered this before specific to the fact that VM SWAP objects are configured with an Object Space Reservation of 100%. We introduced the Sparse Swap setting in 6.2 back in March of 2016.

I wrote a blog post with some PowerCLI code showing how to easily use this new feature here: https://blogs.vmware.com/virtualblocks/2016/02/24/vsan62-powercli-sparse-vswp/

Going a bit deeper than the Object Space Reservation rule, VM SWAP objects are always mirrored (FTM=Mirroring) and adhere to traditional Fault Domain placement rules.

Put simply: Regarding vSAN Stretched Clusters, VM SWAP objects will always have one replica in each site, and a Witness component residing on the vSAN Witness Host.

vSAN 6.6 Stretched Cluster Behavior Using Local Protection

When using a SPBM Policy that supports traditional protection (across sites) as well as Local Protection , the VM objects (VM Home/vmdks/etc) will be mirrored across sites, but will be either mirrored or erasure coded (depending on policy) within the Preferred and Secondary site. Whatever SFTT+FTM is set to, both sites must be capable of satisto be in compliance.

Here’s the same VM using a Local Protection policy, while being protected across sites:

There is a Witness component on each site (part of the Secondary Failures To Tolerate rule) and one on the Witness Host.

vSAN 6.6 Stretched Cluster Behavior Using Site Affinity

When using a SPBM Policy that provides local protection only, the VM objects (VM Home/vmdks/etc) will be either mirrored or erasure coded (depending on policy) within the Preferred or Secondary site. This is an or situation, because the Affinity rule will determine whether the VM’s data resides in one site or the other.

Here’s the same VM using a Site Affinity policy, while being protected in the Preferred site:
Notice that the only Fault Domain listed is the Preferred Fault Domain.

But what about the VM SWAP Object when using vSAN 6.6 with either Local Protection or Site Affinity?

Good question. Remember that the VM SWAP Object doesn’t have a policy applied. It is hard coded. Looking at the VM with the Site Affinity policy applied, what is expected from the VM SWAP Object?
That’s right, the VM SWAP Object still lives on both the Preferred and Secondary sites.

Summary
Regardless of vSAN version (6.1-6.6), VM SWAP Objects will still be written across sites when using Stretched Clusters. This is because the VM SWAP Object has a hardcoded policy.

• For vSAN 6.1 – 6.5 the policy is:
  • Number of Failures to Tolerate = 1
  • Failure Tolerance Method = Mirroring
  • Object Space Reservation = 100%
    *This can be changed using the /VSAN/SwapThickProvisionDisabled advanced parameter
• For vSAN 6.6 using either Local Protection or Site Affinity, the policy is:
  • Primary Failures to Tolerate = 1
  • Failure Tolerance Method = Mirroring
  • Object Space Reservation = 100%
    *This can be changed using the /VSAN/SwapThickProvisionDisabled advanced parameter

**Note, I’ve got a code snippet that will set SwapThickProvisionDisabled for vSAN Clusters located here:
https://code.vmware.com/samples/1665