Configuring EMC Isilon SmartConnect – Part I: SmartConnect Basic
I posted an article on VMware vSphere and EMC Isilon (VMware vSphere and EMC Isilon – One datastore or many?) and covered some of the basics of SmartConnect when used with NFS datastores. I talked about some of the abilities of SmartConnect, but didn’t really cover how to configure it.
As mentioned in my previous post, SmartConnect Basic handles Round Robin IP distribution as nodes are added to a cluster, while SmartConnect Advanced handles advanced IP distribution as nodes are added, removed, or are unavailable, to the cluster. In addition to managing IP addresses for a cluster, through DNS delegation, SmartConnect provides IP addresses for address resolution when connecting to a FQDN, rather than an IP address.
SmartConnect Basic is included with the OneFS operating system without an additional licensing requirements. SmartConnect Basic does provide the ability to use a DNS round-robin connection policy to distribute connections to all nodes in a SmartConnect Zone.
Additionally, as nodes are taken offline for maintenance, or in the event of a failure, are no longer made available from the SmartConnect Zone. This is not a high availability mechanism, but rather a process that indicates that a node, or IP address, is no longer available to answer requests. To truly leverage the round-robin benefits of SmartConnect Basic, clients have to continually perform DNS look-ups to determine if a destination node is available.
Configuring SmartConnect Basic
Configuring SmartConnect Basic is very straightforward. SmartConnect is a network component, click on Cluster > Networking from the Isilon Action menu in the Administrative Web Interface.
By default, a cluster will have at least one subnet, denoted as subnet0 here. As can be seen, I have a SmartConnect zone of cluster.isilon.jasemccarty.com and a DNS of 172.16.1.1.
Clicking on the subnet0 hyperlink will allow us to configure subnet0 and the SmartConnect behaviour.
A couple things to take notice of are:
- Pool0 IP range: 172.16.1.11-172.16.1.13
- SmartConnect settings
- Zone name: cluster.isilon.jasemccarty.com (need this for DNS)
- Connection policy: Round Robin
- SmartConnect service subnet: subnet0
- IP allocation method: static
- Pool members: Nodes 01-03
An important thing to remember about the IP range, is that there does not have to be a 1-to-1 correlation between nodes and IP addresses. I’ll get into that more later, and even more in Part II. Also, SmartConnect Basic will only allow a single SmartConnect zone, while SmartConnect Advanced will allow more.
The SmartConnect settings are where you differentiate between SmartConnect Basic and SmartConnect Advanced.
Because SmartConnect Advanced requires an additional license, SmartConnect Basic is limits the Connection policy to Round Robin, and the IP Allocation method to Static.
A nice warning banner lets us know that additional options require the SmartConnect Advanced license.
After making all of the configuration settings, we need to confirm SmartConnect Basic is working. To do this, open a cmd prompt and ping cluster.isilon.jasemccarty.com.
We will see that we cannot resolve cluster.isilon.jasemccarty.com. This is because we do not have any DNS entries for cluster.isilon.jasemccarty.com.
Remember, that SmartConnect is going to hand out different IP addresses from nodes in our cluster in a Round Robin fashion. With that being said, we can’t simply make DNS A records to point to individual nodes in the cluster. That would defeat the purpose of SmartConnect.
What we are going to do instead, is make a DNS Delegation for the SmartConnect zone (cluster.isilon.jasemccarty.com). When a client queries their DNS server, the DNS server will delegate the DNS lookup to the SmartConnect Service IP. The Isilon cluster will then service the query based on the Connection policy configured for the SmartConnect zone.
In our DNS Management interface, we need to make a New Delegation.
The delegated FQDN is our SmartConnect zone name, or cluster.isilon.jasemccarty.com in this case.
I previously created a DNS A record, called smartconnect.isilon.jasemccarty.com, for my SmartConnect Service IP (172.16.1.10). In the New Delegation Wizard, we can use either a FQDN or an IP address for the name server (SmartConnect Service IP in this case).
In our Windows command prompt, we can now successfully ping cluster.isilon.jasemccarty.com. By pinging the Isilon cluster 3 times, we see the Round Robin Connection policy moving the FQDN from one IP address to the next.
I created a PowerShell script to perform multiple DNS requests and display the output for some additional testing.
Write-Host "PowerShell Script to Query DNS Name/IP distribution"
Write-Host ""
#Clear any DNS cache
Write-Host "Clear DNS cache"
invoke-expression "ipconfig.exe /flushdns"
#Set the SmartConnect zone name
$SCZONE = "cluster.isilon.jasemccarty.com"
Write-Host "Query DNS for IP address of $SCZONE"
#Loop 12 times & perform NSLOOKUP of the SCZONE
ForEach ($number in 1..12) {
try {
[System.Net.Dns]::GetHostEntry($SCZONE)|Select-Object Hostname, AddressList
}
catch {
Write-Host "No DNS Name $SCZONE" -foregroundcolor "red"
}
Start-Sleep -s 2
}
When we execute the PowerShell script, we see the Round Robin Connection policy in action.
It is important to remember, that with SmartConnect Basic, only available IP addresses will be given out. With SmartConnect Basic, IP addresses are assigned 1-to-1 with nodes in the cluster. Node 1 will get 1 IP address, Node 2 will get another, and so on. If a node, and resulting IP address, goes offline, that IP address will not be handed out any longer.
When Node 2 is returned to service, the IP address will be handed out again.
SmartConnect Basic
SmartConnect Basic is a pretty cool feature that is included with OneFS. It brings value by distributing a load across many IP addresses that are respectively assigned to multiple nodes. It is intelligent enough to know which nodes are available and take their IP address out of the list of available IPs when they are offline from scheduled maintenance or an unplanned outage.
If you are attending EMC World 2012, come by the Hands On Labs and take HOL16 – Isilon Setup, Scaling, and Management Simplicity to have hands on experience with SmartConnect.
If you would like to know more about SmartConnect Advanced check out Configuring EMC Isilon SmartConnect – Part II: SmartConnect Advanced.
Hi , Need info and screen shots for windows clusters from basic
Thanks all
Nag
You probably want to reach out to your EMC Rep, or an EMC partner in your area, to see about getting some hands on time with an Isilon cluster.
I’m having an issue and I was wondering if anyone had seen it… I have a DNS delegation set up with a non-user friendly name which points to the SmartConnect IP. We have a “user friendly” CNAME that points to the SmartConnect zone name. Herein lies the problem. There seems to be some sort of disconnect with the CNAME to Delegation, as our users are getting sporadic “access denied” when trying to hit the CNAME. If they go directly to the SmartConnect zone name, all works fine. This is a major issue as all of our windows users home folders are mapped to this particular CNAME (we do this to make back-end changes, like moving to the Isilon, easier).
Any suggestions?
Hmmm…
Not 100% sure I’m following the flow… Let me see if I have this correct…
DNS Delegation setup with a non-user friendly name -> SmartConnect IP (Really should be called SmartConnect service IP)
A user friendly CNAME pointing to the SmartConnect Zone Name
When users connect to the CNAME, they intermittently get disconnects/access denied trying to connect to it.
If that’s correct, then let me see if I can address some of your questions.
1. The Delegation name (cluster.isilon.jasemccarty.com in my example) has to be the same name as the SmartConnect Zone name
2. The delegated DNS server should be the SmartConnect service IP or FQDN of the SmartConnect service IP (172.16.1.10 in my example)
3. Once a Windows client connects to the Delegation name, it will handle authentication, and make a connection.
4. Remember that SMB (1.x/2.x) is a stateful protocol. SmartConnect does not accommodate for any protocol satefulness, only for name resolution and load balancing. As different nodes answer for the Delegation name, this would be indicative of Windows hosts connecting to different nodes and having to authenticate again.
Does this occur with any particular frequency? I mean, does it happen many times in a few minutes?
Not having a better view of your environment, I would really defer to EMC Isilon Support. They can help you troubleshoot the issue better than I can via this blog site.
I’m going to have to be honest… I went round and round trying to get SmartConnect working correctly for about a month (between doing my regular job & not being given any support from our official support organization)…
Thank you,
Jase
Hi Jase,
We have a new cluster online and been having some confusion/issues and various different answers from isilon but since we’re already live and in production can’t really make trial/error changes to the config.
What we’re seeing is when going to https://fqdn.isilon.cluster:8080 to manage the cluster we get logged out of it after a couple of minutes. Connecting via any of the IPs (to the nodes directly) does not log us off.
As far as our DNS/smartconnect configuration we have the following:
SmartConnect zone: cluster.company.com
Delegation name is same as SmartConnect zone name: cluster.company.com
Cluster name: NYCCLUSTER (added to Active Directory) and we have an A record for this name pointing to the smart connect IP
Here is where I’m confused. Our FQDN in the delegation properties is cluster.company.com and IP is the smart connect service IP. We were told by support that this is correct configuration. Reading through your blog post I see that you have a different A record for smart connect pointing to the same smart connect IP called smartconnect.isilon.jasemccarty.com
So I’m wondering if I should A) create a new A record something like smartconnect.company.com pointing it to smart connect IP and use that FQDN in the delegation like the way you have it or B) use our cluster name A record of NYCCLUSTER.company.com
I’ve brought this up to Isilon support and had it escalated and was told to leave everything as is. However, multiple people on that call with isilon had different opinions of what the configuration should look like so I am not so confident with the answer that we got.
Eugene,
Sorry for the delay getting back to you. I am not familiar with an issue of the web interface timing out when connecting to the SmartConnect Service IP.
Isilon support is correct in stating that your setup is properly configured. With that being said, in my example, I’m using a FQDN that is pointing to the SmartConnect Service IP. Keep in mind, my SSIP does not change. The IPs being serviced by SmartConnect do change, but the SSIP does not. Also, the FQDN for the cluster and the FQDN for the SSIP are different (and always should be).
I have configured SmartConnect delegations using both the FQDN for the SSIP as well as the IP of the SSIP. Either “should” behave the same.
As far as “what is correct” I have to refer to support’s recommendations.
Am I correct in stating that the primary issue you are having is that the web interface times out? Or are there more significant issues?
Thanks,
Jase
I also went ahead and configured A records for each individual IP addresses that are part of my IP address pools, probably not required but i like when i run “dig” it returns proper A record.
I’m not sure I would have done that (not that you can’t)… It really all depends on how you want to handle connections/failover/etc…
Tying connections to a specific node IP should be fine when using SmartConnect Advanced and Dynamic/Automatic IP allocation, but it doesn’t really tie into load balancing the cluster as a whole. That’s really where the SmartConnect Zone name comes into play. If you are referencing a single IP, then the connection will ONLY be to a single node. There is no way around that.
Again, with SmartConnect Advanced and Dynamic/Automatic IP allocation, in the event of maintenance or a node failure, that IP will be reassigned to another node. I couldn’t elaborate as to whether the transition would be seamless for NFSv3, as it is for the SmartConnect Zone.
Also, keep in mind, if you are using this storage for a vSphere NFS mount, if the DNS name or IP are different for a datastore, even though the NFS share resides on the same cluster, it will be treated as a DIFFERENT datastore. This is because the path (either individual IP/DNS name) is different for each.
Connections that are tied to the SmartConnect Zone (even with different IP addresses) will be presented as the SAME datastore. This is because the path (SmartConnect Zone FQDN) is the same.
Make sense?
@jase
Hi Jase,
Thanks for your reply.
The web interface timeout is our primary issue. It seems to be limited to Chrome and Firefox but not IE so not fun for my colleague who uses a Mac as a primary machine, he just connects to an IP of one of the nodes. We are using the cluster mainly for SMB fileshares. I’ve carved out a share for some of my Veeam backups and initially was having issues where jobs would just hang every few days. That hasn’t happened in the last few weeks so maybe not related to the timeout issue which is still happening as of a few minutes ago.
We did work with Isilon on resolving some additional issues with jobs failing and delayed notifications of cluster events to isilon support. They were being caused by failing out 3 cluster nodes. You see we had 3 nodes for our initial POC and started using them for production. Once we received our production nodes they were added to the cluster and then original 3 nodes were failed out. For some reason this didn’t happen as cleanly as it should have. Currently everything looks good with our cluster except for timeouts of Chrome and FF. I haven’t had a chance to follow up with Isilon. Last time I spoke with them they suggested updating java on our computers though didn’t have any specific version recommendations.
We are still on vSphere 4.1 and SmartConnect is not supported. Everytime you rescan in vSphere, the NFS datastorage would up multiple times with a (1) in parentheses. You mention SmartConnect fo VMware datastores but if i look at any Isilon best practices they are not using it, they manually load balance each ESX node to every node in the cluster.
VSphere 4.1 is definitely different.
The latest info I can point you at is, a Reference Architecture from April of 2012:
http://www.emc.com/collateral/hardware/technical-documentation/h10680-ra-isd-and-vsphere-4.1.pdf
i am sure 5.x is SmartConnect friendly but last month’s presentation by Swaroop Dutta still has the same slide where each ESX host is connected to every single isilon node, bypassing SmartConnect.
I’ll reach out to some of the Isilon resources I know, and I’ll get an update for some clarity.
Update: I’m assuming you are referring to this graphic:
http://www.jasemccarty.com/blog/wp-content/uploads/2012/04/ISI-POST02.png
With this configuration there are some things to note:
*Each host is connected to each IP (depending on the IP allocation policy, this could mean 1 or more IP address per node)
*Each IP/NFS Mount will show up as a different datastore.
Yes, a mountpoint of /ifs/nfs1 would show up different for each IP/NFS Mount combination in vSphere.
*Gives you the ability to manually manage which VMs are on which datastore (IP/NFS Mount combination) from a front end pespective
So VMs can be designated to connect on particular nodes (depending on the presented Node IP address).
This configuration does give you additional control of the placement of VMs, particularly when it comes to which IP/NFS Mount the VMs run off of. SmartConnect Advanced, when used with Dynamic IP Allocation, provides value in this situation, because when a node is taken offline for maintenance or suffers a power or hardware failure, the IP moves to another node. This is a manual process, and is independent per datastore IP/NFS Mount. This method provides granular control of VMs on the hosts, because control is per IP/NFS Mount, and not per host. This works for vSphere 4.x and 5.x.
Using the SmartConnect Zone name (FQDN), only supported by vSphere 5, dedicates an ESXi host’s traffic for the datastore (FQDN/NFS Mount) to a single IP address. Because all of the VMs use the same FQDN/NFS Mountpoint, and the host resolves a single IP address, all VM traffic for that datastore is sent to the same IP address/path. Again, this works on vSphere 5.x. Keep in mind, if you want to move a VM’s path (different Node IP on the backend), you would have to move it to another host that has that IP resolved for the FQDN/NFS Mount. This would only require a vMotion. The previous method would require a Storage vMotion to move it to the different path (because it is registered on a different datastore [IP/NFS Mount]).
Keep in mind, that the Dynamic IP allocation portion of SmartConnect Advanced, provides the most important part of both configurations, especially when a planned/unplanned outage occurs, by moving the IP to another available node. This happens, regardless of whether you are leveraging the SmartConnect Zone name (FQDN) or not.
Which is correct? They both are correct (for 5.x), it all depends on what level of control or automation or path flexibility you want to have.
And of course, these opinions are mine, and not necessarily those of EMC or Isilon.
>>> This configuration does give you additional control of the placement of VMs, particularly when it comes to which IP/NFS Mount the VMs run off of. SmartConnect Advanced, when used with Dynamic IP Allocation, provides value in this situation, because when a node is taken offline for maintenance or suffers a power or hardware failure, the IP moves to another node. This is a manual process, and is independent per datastore IP/NFS Mount .>>
what do mean this process is manual ? I am using dynamic ip allocation for my subnet that i use for NFS clients (including VM). If a node happens to panic or i am doing rolling upgrade, its ip addresses move to another node in the cluster automatically. While i can move IP addresses from one node to another (pool that uses dynamic ip allocation) i typically don’t have to do that unless i manually want to move some stuff around.
<>
while this approach gives my VM admin more control on VM placement, it also creates more confusion because they see the same file system/datastore listed 6 times (i have 6 nodes), same utilization so it’s not very intuitive.
What I meant by additional control was…
If VM1 is registered on IP1/NFS Mount, then it will always use the node IP1 is assigned to.
If VM2 is registered on IP2/NFS Mount, then it will always use the node IP2 is assigned to.
This is independent of which host VM1 or VM2 are currently registered on.
When using the SmartConnect Zone name (FQDN)//NFS Mount, all VMs on a host will use the same IP that the host resolves after DNS resolution of the FQDN.
And yes, if 1 NFS mount is registered with different IP addresses, things like the total amount of storage are not reflected properly, much in the same way as mounting 2 or more network drive mappings on a single Windows server report the same amount of free space.
You might want to reach out to your Isilon Technical Consultant or Account Representative for some additional resources particular to your implementation.
this is how it was configured and blessed by the local team. I am just pointing out how clunky it is from customer perspective
Quick question. When I set up the smartconnect per your example above, pinging the zone name responds with the SSIP instead of going through the round-robin way!
Keep in mind that the SmartConnect Zone name must match (exactly) the DNS Delegation address.
I just setup an Isilon 6.5.5.11 cluster, with the following settings:
ssip65.vlab.jasemccarty.com – 192.168.10.180
DNS Delegation for onefs65.vlab.jasemccarty.com points to ssip65.vlab.jasemccarty.com
IP range for the cluster is 192.168.1.181-192.168.1.188
Performing a DNS query of ssip65.vlab.jasemccarty.com returns 192.168.1.180 (this is the IP that ANSWERS the DNS request, and gives out IPs from the Pool [Pool0 in this case])
Performing a DNS query of onefs65.vlab.jasemccarty.com returns 192.168.1.181-188 changing every time.
I honestly struggled with the config at first (when I first did this) and it took a few tries to get it right.
1. DNS Delegation is the key
2. DNS Delegation points to the SmartConnect Service IP (192.168.1.180 in this situation)
3. Pool0 which is part of Subnet0 has a range of 192.168.1.181 to 192.168.1.188
4. When querying the SmartConnect Zone Name, it will return the IPs in the Pool.
With or without a SmartConnect license this will work (Basic is free, Advanced is an additional charge)
I hope this helps, and let me know how it goes.
Jase, thanks for the quick reply. It’s amazing how simple this stuff seems and how complicated it can be in real life. Or maybe it’s just my little brain
Hmmm…so I’ve got this config and I see where your example and my config differ:
smartconnect zone: isilonx200.domain.com
SSIP: x.x.x.100
pool range: x.x.x.101-103 (3 nodes)
DNS delegation: isilonx200.domain.com points to the .100 (SSIP)
When you say: DNS Delegation for onefs65.vlab.jasemccarty.com points to ssip65.vlab.jasemccarty.com <– what is "onefs65"? Is that the cluster name? I'm also struggling with cluster name vs zone name. If the cluster is named XYZISILON, can the zone name also be XYZISILON.domain.com?
Jase, never mind. I did figure it out. It was not working because I had the DNS delegation to the SSIP and did not create and host entry and pointed to it.
Good! Sorry for the delay… Responses are unfortunately sporadic given “day job” availability.
I hope that my instructions provided some help.
Hi Jase,
Nice article. I understand from the forward lookup using this delegation method, is there any gotchas when doing reverse lookup using IP address of cluster node which is using RR with static method? In you case, if user ping using 172.16.1.13 will the resolution to cluster.isilon.jasemccarty.com happens?
Actually, you will not get an address name resolved, as the IPs aren’t tied to the SmartConnect Zone’s FQDN in a persistent fashion, but rather based on the SmartConnect Basic/Advanced policies.
From a Forward Lookup perspective, DNS delegates to the SmartConnect Service IP. From a Reverse Lookup perspective, that information isn’t contained in a Reverse Lookup Zone as part of a normal configuration.
I tried this in my lab (adding a reverse delegation), and you would think that you could perform a nslookup against the SmartConnect Service IP, but unfortunately it does not work their either.
The reason it won’t work if performing an nslookup against the SSIP, is that the SSIP is configured only to answer requests for the SmartConnect Zone name. An IP address isn’t going to match, and as a result, a query of this type is going to be refused.
I hope that helps.
Jase
Jase,
Thanks for the article. I have one question which is bugging me for some days now. Why do we need a zone delegation at all? Why cant we just have an ‘A’ record. In the example, you have a zone called -”cluster.isilon.jasemccarty.com”. Why cant we create an A record for this with the IP address of SSIP? Why delegation required at all? (may be this is a stupid question. But please explain).
Sudheep,
A couple things you have to remember about this setup are:
1. The SmartConnect Service IP (SSIP) will always reside on the lowest node (that is available) in the cluster (Node1 in most cases). This IP does not move between nodes until the lowest node number goes offline (planned/unplanned).
2. The SmartConnect Zone DNS Delegation is provides the process where the zone “cluster.isilon.jasemccarty.com” (in this case) is then resolved by the SmartConnect engine on the cluster.
3. The IP address is then returned based on the SmartConnect license (Basic/Advanced) and Connection Policy.
4. SmartConnect Advanced provides greater flexibility around choosing which nodes to use for the common name (SmartConnect Zone).
Without a DNS Delegation, and only using an A record, the FQDN would always point to the lowest numbered node. In this situation, it would be similar to simply providing a static IP to a node, and pointing a FQDN to that node. That process defeats the single namespace, scale out benefits of Isilon & OneFS from the perspective of adding processing, throughput, and physical interface perspectives. The scale out file system would still be of benefit, but would only be accessed through the single referenced node.
The DNS Delegation process, in conjunction with SmartConnect Advanced, gives the ability to not just scale out the file system, but also the amount of processing power, amount of total throughput, and count of physical interfaces.
I hope that provides a little more clarity.
Jase
I am trying to set EMC Isilon on ESXi and the problem I am having is the range of ip Addresses for int-a
I have two Nics, the one is set to vSwitch that is not connected to any physical nics.
Since it is a Esxi server, there are no VMnet1 and 8..
How do i figure out the range of Ips???
Khan,
Am I correct when I ask if you are talking about a virtual Isilon node?
Jase
Yes, I just started setting it up!!! and I have created a new switch that is not attaches to any Nics so I can use it for Host only.. and I am not sure what IP range it would accept
Nevermind.. Just figured it out!!! incase some1 netmask is 255.255.255.0 and ip range is 10.0.0.1 to yay many nodes!!!!
This really isn’t the appropriate forum for conversations around the virtual node, but I will say… vNIC0 is the back end and vNIC1 is the front end.
Beyond that I’m going to have to recommend consulting the source that you received the VSA from.
Additionally, it is not supported as a production storage device.
Good luck with it!
Good Morning Jase,
Would you please share your take on EMC’s own private network 128.x.x.x for the internal subnet (infiniband switches)?
What will be the benifit of doing so?
Thanks & I love your blog:>)
Henry
Henry,
My take? That’s a good question. Having only been involved with Isilon storage since coming to EMC, I am honestly still learning.
I’m going to reserve my comments for the internal subnet, other than to say I’ve seen a couple different config recommendations for the addressing. When I attended an official Isilon training course, we used 10.10.10.x for Int-A, and 10.10.20.x for Int-B, with the virtual/LB range being 10.10.30.x.
I’ll see what I can dig up about 128.x.x.x for the internal subnet.
Thanks for the kudos, and I’m glad I am bringing some value!
Cheers,
Jase