Configuring the EMC Isilon NAS VAAI Plugin for vSphere 5.0/5.1

Isilon OneFS 7.0 has been out for a couple months.  OneFS 7.0 has brought some additional features to Isilon, particularly toward VMware vSphere support.

I blogged about VASA support in another article: Configuring the EMC Isilon VASA Provider.

This article is going to focus on how to configure the NFS VAAI Plugin for Isilon when used with VMware vSphere 5.

Requirements
Here are the basic technical requirements to get the plugin working:

• An Isilon cluster running OneFS 7.0.0.0 or higher
• VMware vSphere 5.0
  • I have used vSphere 5.1 without issue
  • *I have heard of several situations where 5.0 does not work successfully
    until upgraded to 5.0 Update 2
  • **I have not seen this work with vSphere 5.5 (as of early September 2013)
• A SmartConnect zone configured for the presented datastores
• NFS mounted datastores being presented from the Isilon cluster using the SmartConnect zone name (can be short or FQDN)
• All IP addresses in the Isilon pool must be “Allowed IP addresses” in the NFS Client settings in ESXi

Installation
The installation process is not 100% straight forward according to the Release Notes, which require a support.emc.com login to get to.  I’ll try to provide a little clarification here.

The expected process is as follows:

1. Enable the vCenter for Isilon Service on the Isilon Cluster by running isi services isi_vc_d enable from the Isilon command prompt.
2. Load the plugin on the ESXi host & configure authentication

In fact there is a little bit more to do.  Enabling the vCenter for Isilon Service doesn’t have any requirements that I’m aware of, other than to run the command, but loading the plugin and configuring authentication can be tricky.

Loading the plugin
For starters, after downloading the Isilon NAS VAAI Plugin (VMW-ESX-5.0.0-isilon_nas_vaai_plugin-1.0-offline_bundle-834058.zip), adding it to VMware Update Manager, and creating a baseline, I noticed that despite the fact that the plugin had not been installed on my hosts, it appeared to have been.

I wasn’t able to push the extension to the hosts using VMware Update Manager.

I have only been successful in manually installing the extension on my ESXi hosts, but that isn’t that straightforward either.

I uploaded the VAAI plugin files named “VMW-ESX-5.0.0-isilon_nas_vaai_plugin-1.0-834058.zip” and “VMW-ESX-5.0.0-isilon_nas_vaai_plugin-1.0-offline_bundle-834058.zip” to my datastore and attempted to load both of them. No dice.  Not until I extracted the vib (isilon_nas_vaai_plugin.vib) did I see success.
*Note whether EMC_bootbank_isilon_nas_vaai_plugin_1.0-0.vib is extracted from the offline bundle, or isilon_nas_vaai_plugin.vib extracted from the other, the vib that gets installed is still named EMC_bootbank_isilon_nas_vaai_plugin_1.0-0.vib.

The installation command (with the vib loaded in my local datastore1) is:
esxcli software vib install -v=”/vmfs/volumes/datastore1/isilon_nas_vaai_plugin.vib”
*Note the Release Notes state a requirement to set software to CommunitySupported, this is not required as the Isilon NAS VAAI plugin is signed as VMwareAccepted.

Configuring authentication
After the vib is installed, authenticating to the Isilon cluster is required.

To configure authentication, run the following command:
/usr/lib/vmware/nas_plugins/isi_plugin_credential/auth_gen

Information required includes the SmartConnect zone name, user account, and password that will be used to connect to the cluster.
*Note: The Release Notes do not mention requiring the SmartConnect zone name.  I brought this up to the team during the Alpha release, with the argument of having multiple Isilon clusters in a single environment, and the lack of a specified SmartConnect zone would require all clusters being connected to, to have the same credentials.  I’m not certain if I had anything to do with the addition, but I’m glad to see it made it into the 1.0 release.

When running this the first time, I got an error.

Internal Error S7, please contact EMC Isilon Support for details
, addr is https://mavericks.vlab.jasemccarty.com:8080/platform/1/cluster/external-ips
failed to connect to https://mavericks.vlab.jasemccarty.com:8080/platform/1/cluster/external-ips, with status 7.
please check if the host you are trying to reach is a valid Isilon Storage cluster, ensure that Apache service is up
Internal error FM001, please contactEMC Isilon Support

Ok… Where do I start?

I added the datastore to my ESXi host…

Still no success.

Internal Error S7, please contact EMC Isilon Support for details
, addr is https://mavericks.vlab.jasemccarty.com:8080/platform/1/cluster/external-ips
failed to connect to https://mavericks.vlab.jasemccarty.com:8080/platform/1/cluster/external-ips, with status 7.
please check if the host you are trying to reach is a valid Isilon Storage cluster, ensure that Apache service is up
Internal error FM001, please contactEMC Isilon Support

Same error, but I knew I could “reach” the cluster, as I had it mounted… Then I tried the command again…

There it was… What was the difference?

Remember that when connecting to an Isilon cluster using the SmartConnect zone name, different IP addresses can be used when I request is sent to the SmartConnect service IP?

My cluster has 3 IP addresses.

When ESXi connects to an NFS mount, the IP address of the NFS mount (using a name or IP) is then added to the NFS Client’s settings in the ESXi firewall.

Looking at my NFS Client settings I found that only 2 of my node IP addresses had been added.  Without adding a lot of redundancy to this post I’ll share that the primary requirement is that all of the IP addresses of the SmartConnect zone being connected to, must be in the NFS Client “allowed connections” list.
*If all of the IP addresses are not listed, auth_gen will need to be run several times until the process is executed against an IP that is allowed

Something else I found, was that if I rebooted my ESXi, when the Isilon NFS share is mounted, if the IP that is returned by SmartConnect isn’t already in the “allowed connections” list, it is added and the datastore is still mounted.

So my datastore is mounted, the Isilon NAS VAAI plugin is configured, I should be good to go… I look at my storage, and I see…

Hardware Acceleration is not supported. What have I done wrong?

Well, the release notes state that no reboot is required after installing the plugin.  After a reboot, I saw success.

Summing it up
To get the Isilon NAS VAAI Plugin working, I would suggest the following steps:

1. I’m not certain this is the process, but reach out to your Isilon SE/Rep and request the plugin
2. Enable the Isilon Service for vCenter
   (isi services isi_vc_d enable) It doesn’t appear this is required
3. Attach the Isilon NFS mount on your ESXi host(s)
4. Upload the vib (isilon_nas_vaai_plugin.vib) to the Isilon datastore seen by your host(s)
5. Install the plugin
   (esxcli software vib install -v=”/path/to/plugin/isi_nas_vaai_plugin.vib”)
6. Run the auth_gen application from the ESXi console
   (/usr/lib/vmware/nas_plugins/isi_plugin_credential/auth_gen)
7. Reboot the ESXi host

After quite a bit of troubleshooting, I don’t believe it is a requirement to have the datastores mounted beforehand (after a conversation with Larry Orloff), but the requirement for all the IP addresses of the SmartConnect zone being present in the “allowed connections” and a host reboot were consistently required.

I reached out to the folks that developed the plugin with a couple more questions about automating this process.  In speaking with them, unfortunately in 1.0, the auth_gen process requires interactive inputs, and does not accept parameters.  I’ve requested that functionality be added.  Not sure if/when it will, but I’m optimistic it will at some point.

Things to note
I have been using the plugin with vSphere 5.1 without issue.

There is an ECN thread where a user reported an issue with some vSphere 5.0.0 hosts, regarding the plugin not behaving properly.
That thread can be found here: https://community.emc.com/message/711444

Also, VMware has a KB article states some issues with the plugin when using vSphere 5.0.x.That KB article can be found here: http://kb.vmware.com/kb/2042590

A couple updates
After this post published today, I reviewed the ECN article above, and took the opportunity to stand up a new ESXi 5.0 Update 1 host.

This is what I found when using vSphere 5.0:

• First Try – Unsuccessful:
  • Installing the plugin before any patches were applied, and rebooting the host resulted in some connectivity issues.
  • I could ssh in, but couldn’t manage the host from vSphere.
  • I also could not run esxcli from an ssh session, and could not access the ESXi shell from the host’s console. (unsuccessful)
• Second Try – Successful:
  • After reinstalling ESXi 5.0 Update 1 and adding it back to vCenter, I could apply, and deploy the plugin using VMware Update Manager
  • I installed all the relevant patches for ESXi 5.0 Update 1 (updated to build 914586) which results in ESXi 5.0 Update 2.
  • Rebooted
  • Installed the Isilon plugin with VMware Update Manager
  • Rebooted
  • Attached my Isilon NFS mount
  • Configured auth_gen
  • Rescanned my storage (successful)
• Third Try – Successful then Unsuccessful:
  • I stood up a new ESXi 5.0 Update 1 host, added it to vCenter and I could apply and deploy the plugin using VMware Update Manager
  • Installed the Isilon plugin using VMware Update Manager (as in my first attempt)
  • Rebooted
  • Attached my Isilon NFS mount
  • Configured auth_gen
  • Rescanned my storage (successful)
  • I installed all the relevant patches for ESXi 5.0 Update 1 which updated it to 5.0 Update 2 (build 914586)
  • Host no longer available as in my first attempt (unsuccessful)
• Fourth Try – Successful:
  • Upgraded the ESXi host in attempt #3 to ESXi 5.1.0 GA (build 799733) and everything worked properly (success)
  • Installed all patches to the 5.1.0 GA host bringing it up to 5.1.0 Patch 1 (build 914609) and everything continued to work properly (success)

With that being said, if you are running vSphere 5.0, I’d recommend you upgrade to at least ESXi 5.0 Update 2 (build 914586) before installing the plugin.  If you run into any issues, I’d recommend contacting EMC Support.
Additionally it would not appear there are any issues with vSphere 5.1.

Update (3/15/13): I was looking at the EMC Community Network, and noticed this post: ISILON VAAI plug-in: auth_gen Segmentation fault
If you have a root password that is 8 characters, or divisible by 8, you might receive a Segmentation fault.  It is mentioned in the post that it should be corrected in the next release of the plugin.

Update (9/6/13): I was working with one of our Support Engineers on a case (Thanks Paul D.) and the FM001 error appears to occur whenever there is a communications failure with the FQDN (remember this changes with each request).  Above I mentioned that you want to pre-authorize all IP addresses of your pool in the NFS Client settings of the vSphere Firewall rules.  Make sure there is connectivity on port 8080 between your host and the SmartConnect Zone/Pool.  A simple method to check this is using this command: nc -z SmartConnectZoneName 8080

You should receive something like this: Connection to SmartConnectZoneName 8080 port [tcp/webcache] succeeded!

If you don’t get that, it isn’t going to work.