by Looking back that the SmartConnect Part I post, remember that SmartConnect is handling DNS delegation of the SmartConnect Zone name (FQDN)… So… What happens if the SmartConnect Zone for an NFS data network is on an isolated network? Keep in mind, that DNS needs to contact the SmartConnect Service IP to request an IP resolution for the FQDN. How can this happen if the SmartConnect Service IP for that zone is on an isolated network?
The first network defined for the “front-end” is typically configured as subnet0. It is a common practice to leverage subnet0 for the management of an Isilon cluster. When using IP based storage in vSphere environments, it is a best practice to have the storage traffic isolated from the regular network. In my previous posts (SmartConnect Part I & SmartConnect Part II) the SmartConnect Zone was on the same subnet/pool. When this is the case, and isolated networks are being used, it is impossible to leverage SmartConnect on the isolated network, unless there is a DNS, and subsequent delegation, in that isolated network.
In my demo environment, I have the following configuration:
- 8 Isilon nodes
- subnet0 configured with
- subnet 192.168.1.0
- netmask 255.255.255.0
- gateway 192.168.1.1
- SmartConnect service IP: 192.168.1.80
- SSIP DNS Name: ssip.domain.local
- pool0 configured with
- IP range; 192.168.1.81-192.168.1.88
- Zone name: isilon.domain.local
- SmartConnect service subnet: subnet0
- subnet1configured with
- subnet 172.16.1.0
- netmask 255.255.255.0
- gateway 172.16.1.1
- SmartConnect service IP: 172.16.1.80
- SSIP DNS Name: ssip.nfs.domain.local
- pool1configured with
- IP range: 172.16.1.81-172.16.1.88
- Zone name: isilon.nfs.domain.local
- SmartConnect service subnet: subnet1
The following graphic details the typical look-up process for the isilon.domain.local SmartConnect Zone.
The process is as follows:
- Query the Isilon Fully Qualified Domain Name (isilon.domain.local)
- DNS delegates the query to the SmartConnect Service IP (ssip.domain.local DNS record)
- SmartConnect gives out the SmartConnect Zone Name
- The FQDN is accessed by the Management Network
Because the IP returned is on the 192.168.1.x range, the data flow is between the vSphere Management Network and subnet0/pool0 of the Isilon Cluster.
As previously stated, this isn’t the preferred method. Subnet0/pool0 aren’t isolated.
We really want our NFS datastores to be accessed on our NFS Storage Network (172.16.1.x) which is serviced by subnet1/pool1 on our Isilon Cluster, with the SmartConnect Zone name of isilon.nfs.domain.local.
Following the same process, it would be:
- Query the Isilon Fully Qualified Domain Name (isilon.nfs.domain.local)
- DNS delegates the query to the SmartConnect Service IP (ssip.nfs.domain.local DNS record)
SmartConnect gives out the SmartConnect Zone Name
The FQDN is accessed by the NFS Data Network
But this doesn’t work… Because there is no connectivity from our DNS server to our SmartConnect Service IP for our second Zone (isilon.nfs.domain.local).
Examining at the SmartConnect settings for each pool will show why this does not work.
Pool0
Pool1
Notice how each pool is leveraging the SmartConnect service subnet that it resides in?
Pool0, which is part of subnet0, is using the SmartConnect service subnet of subnet0.
The DNS server has access to subnet0, and can properly return a value for isilon.domain.local.
Pool1 on the other hand, is configured for a SmartConnect service subnet of subnet1.
Because the DNS server does not have access to subnet1, it cannot delegate to the SmartConnect service IP (172.16.1.80) for resolution of isilon.nfs.domain.local.
Making a couple changes will correct the issue.
- Update the DNS Delegation to point to ssip.domain.local
(which is on subnet0, and accessible by the DNS server)
- Update the SmartConnect service subnet for subnet1/pool1 to use subnet0 to answer SmartConnect queries on behalf of subnet1.
Now the process is:
- Query the Isilon Fully Qualified Domain Name (isilon.nfs.domain.local)
- DNS delegates the query to the SmartConnect Service IP (ssip.domain.local DNS record)
- SmartConnect gives out the SmartConnect Zone Name
- The FQDN is accessed by the NFS Data Network
Attempting this again, shows the proper results.
This graphic demonstrates the process:
In my previous SmartConnect posts (SmartConnect Part I & SmartConnect Part II), I did not really cover the process of assigning a different SmartConnect service subnet to accommodate for subnets/pools that were isolated. I hope that this clears things up.
And just for refreshers, the Best Practice Guide can be found here: EMC Isilon Scale-out Storage and VMware vSphere 5 Best Practice Guide.
Thanks go to James Walkenhorst for his input.
Update: Something I forgot to add. in the past, I have always setup a gateway address and SmartConnect service IP on each subnet I had created. Upon standing up a new cluster tonight, I realized that the above method also works for subnets/zones that do not have a SSIP associated with it. In standing up a new cluster, I left the SmartConnect service IP blank (not really using it on the isolated network), and it still works. Again, this is because the SSIP for subnet0 is answering for the SmartConnect Zone on subnet1. (I used different addresses, as my primary cluster is still running)
Happy to see SmartConnect service subnet as an option in IP address pools for addressing the isolated network problem typical of NFS traffic. This had been a design pain point in earlier versions.
what would be even better is to be able use multiple service subnets for one zone name. Because with the config you proposed above if you have a system that only resides on 172.16.1.x network, it will no longer able to use smartconnect to connect to the cluster.
Multiple service subnets for a single zone name? Hmmm.
It took me writing (and erasing) a couple responses to see what you meant…
1. Above config in place
2. A system on the isolated network
3. System on the isolated network cannot leverage SmartConnect.
That is correct. As of this writing, only one subnet can answer for a zone.
yep, let me give you another use case:
subnet0- 172.16.1.x (private subnet for NFS connections)
subnet1 – 10.16.1.x (customer 1)
subnet2 – 10.23.1.x (customer 2)
customer 1 and customer 2 networks are firewalled (or not routed). So if i configure subnet0 with subnet1 as my service subnet, great 10.16.1.x customer is happy but customer 10.23.1.x is out of luck.
So you are looking at this from a Service Provider perspective.
The only way that you would be able to make that work, would be to have a DNS server (or multiple DNS servers) that would have access to management subnet (possibly subnet3) for the DNS delegation.
A feature request would be required to add the behavior you are looking for.
i work at a big university with many distinct customers/ network zones ..so sure, i consider myself a service provider 🙂
so you would create subnet3 and use that as my service subnet ? That would require yet another subnet and subnet1 and subnet2 would have to be able to access subnet3.
Just for clarification, this is for isolated VM data traffic correct? Or for file shares?
this could be for VM traffic or simply dedicated network for NFS traffic
The issue is that multiple interfaces/subnets cannot answer for a single SmartConnect Zone name (FQDN). What is preventing the DNS servers on the 10.16.1.x and 10.23.1.x networks from reaching the same (common) SmartConnect Service IP?
If it is truly limitation that cannot be overcome, and these are “truly isolated” networks, why not do something like this:
subnet0 – 192.16.1.x – Isilon Management Network (update to yours)
subnet1 – 172.16.16.x – NFS network for 10.16.1.x network
subnet2 – 172.16.23.x – NFS network for 10.23.1.x network
subnet3 – 10.16.1.x – SmartConnect Service IP network for isilon.sixteen.domain.local
subnet4 – 10.23.1.x – SmartConnect Service IP network for isilon.twentythree.domain.local
Just a suggestion. I understand it isn’t what you are looking for, but sometimes you have to get a little creative to accomplish what you need.
I would highly suggest that you reach out to your Isilon Technical Consultant and/or Account Representative and ask for a feature request to be submitted on your behalf to modify the behavior of SmartConnect to incorporate the feature you desire.
What is preventing the DNS servers on the 10.16.1.x and 10.23.1.x networks from reaching the same (common) SmartConnect Service IP? <<
subnet0- 172.16.1.x – is private subnet and not routable, DNS servers can't get to it.
I understand that the 172.16.1.x subnet (subnet0 in your case) is not reachable.
This is where another subnet (subnet3) would be used. Where, at a minimum, a DNS server that both the 10.16.1.x and 10.23.1.x can reach, as connectivity to subnet3 for SmartConnect zone resolution.
that’s yet another subnet that i have to request just for SmartConnect to work and establish connectivity between two networks that should have no inter-connectivity at all, not likely to happen. We’ll see what my TC says about filling out some paperwork for RFE
I will reach out to some more folks, and see if there is a workaround.
I’ll keep you posted.
thank you Jase
any ideas from your colleges ? Don’t say Maverick 🙂
Unfortunately not yet…
And no, I’m not going to say Mavericks…
I’ve tried this, and it does work. It is so easy to break by simply running nslookups for the smart connect zone you are using to perform the lookups for the isiloated network. It seems to break the Windows DNS server. Deleting the cache on the Windows DNS server resolves the issue. Just wondering if anyone else has seen this issue.