Monday, 25 April 2011

Building a RHEL 6/Centos 6 HA Cluster for LAN Services (part 5)

Clustered NFS Server
I'm now going to add clustered NFS services. I'm going to have a shared projects area and an area for homedirs. To provide rudimentary load balancing I'm going serve these from separate nodes by default. Also as these are going to be using the lions share of my storage I'm going to make them quite large.

I don't know, but I suspect that for a large data area where performance is an issue it's probably not a good thing to grow these in small chunks. I'd assume this might lead to LVM fragmentation which may hurt performance.

First step lets bring up the storage as ever (on one node):

/sbin/lvcreate --size 200G --name lv00home cluvg00

/sbin/lvcreate --size 500G --name lv00projects cluvg00

/sbin/mkfs -t gfs2 -p lock_dlm -j 2 -t bldg1ux01clu:home /dev/cluvg00/lv00home

/sbin/mkfs -t gfs2 -p lock_dlm -j 2 -t bldg1ux01clu:projects /dev/cluvg00/lv00projects

Update fstab and mount these up on both nodes.

Also we need to add a parameter to make statd cluster and failover aware (or at least help it with this, or so I'm told). So on both nodes add to /etc/sysconfig/nfs:


In the cluster.conf file we need to add a single nfsexport (this ensures the daemons are working) resource and an nfsclient resource for each thing we are exporting.

<ip address="" monitor_link="1"/>

<clusterfs device="/dev/cluvg00/lv00projects" fstype="gfs2" mountpoint="/data/projects" name="projectsfs" options="acl"/>
<nfsclient name="nfsdprojects" options="rw" target=""/>                                      
<ip address="" monitor_link="1"/>
<clusterfs device="/dev/cluvg00/lv00home" fstype="gfs2" mountpoint="/data/home" name="homefs" options="acl"/>
<nfsclient name="nfsdhome" options="rw" target=""/>

And then two service definitions for each of these:

<service autostart="1" domain="bldg1ux01A" exclusive="0" name="nfsdprojects" recovery="relocate">
               <ip ref=""/>
                <clusterfs ref="projectsfs">
                     <nfsexport ref="bldg1cluexports">
                           <nfsclient ref="nfsdprojects"/>                                                                           
 <service autostart="1" domain="bldg1ux01B" exclusive="0" name="nfsdhome" recovery="relocate">
                <ip ref=""/>
               <clusterfs ref="homefs">
               <nfsexport ref="bldg1cluexports">
                    <nfsclient ref="nfsdhome"/>

Notice that they are in the different failover domains to direct them to be served by different nodes by default (unless one fails that is).

There is an issue with this and NFSv3 clients. The portmapper replacement in RHEL6, rpcbind, by default it replies to incoming requests using the node IP rather than the service IP. This confuses client firewalls so they fail to mount. The only real work around just now is to fully open up both node IP's on the client Firewalls e.g at the bottom of the client machines /etc/sysconfig/iptables on say RHEL 5

-A RH-Firewall-1-INPUT -s -j ACCEPT
-A RH-Firewall-1-INPUT -s -j ACCEPT

Not great (bz#689589). Sadly the flags on rpcbind that are used for multi-homing , that should help with this don't seem to work when the IP isn't up when rpcbind is started.

Again add to DNS and hosts these service IP's   bldg1clunfsprojects bldg1clunfsprojects.lan   bldg1clunfshome bldg1clunfshome.lan

Bump the cluster.conf version number, verify the file and propagate and you should be able to mount the exports, either in a hard mount or via the automounter on client machine e.g

mount bldg1clunfshome:/data/home /mnt
mount bldg1clunfsprojects:/data/home /mnt


ls /net/bldg1clunfshome/data/home
ls /net/bldg1clunfsprojects/data/projects

Clustered Samba

NOTE: I have recently discovered that Red Hat do not allow you to share a GFS2 filesystem between local access (e.g Samba) and NFS. This is due to a lack of interoperability between local file access locking and NFS file locking (flocks vs plocks). On other filesystems this may lead to file  level corruption, however on GFS2 this may lead to FILESYSTEM level corruption ! And/or kernel panics (flocks, plocks and glocks not getting along, and I'm not making this up!)

So the below notes are fine if you use a filesystem shared out on Samba that isn't shared out on NFS (not as below but a new and different filesystem). Which is very unfortunate, as if you are anything like me it's exactly what you want to do (share files between Windows and Linux).  Or you can share out on samba an NFS mount (what RH say to do) but not recommended by the Samba people. I'm also pretty sure my backup software won't like to backup an NFS mount!

My solution to this that I will document here, was radical surgery. Reimplement my cluster using ext4 failover mounts. I can easily live with not relying on locks working between Samba and NFS (I  never expected this to work), I can't live with filesystem corruption and kernel panics. 

The original documented case here may eventually work once bug #580863 is resolved. If you want to see the rather murky world of Linux file locking this article is great here. I have left the original text unchanged below in the hope this bug gets resolved.

There are two ways of clustering samba. One if to use a failover method in the cluster.conf file. But a new way is to use samba's relatively new built in clustering. This provides load balancing (whereas the failover samba only provides HA). This method is outside the standard cluster.conf, but RH ship it so lets use it.

You need to ensure ctdb package is installed on both nodes.

I'm going to first create a common locking directory, clustered samba uses this to share a locking directory. For my own purposes, outside samba, I also use it to hold locks for cronjobs etc that can run on either node. I'm also going to site my printer click to print drivers for Windows in here.

So for this it's usual deal and setup (with some added setups for the samba areas):

/sbin/lvcreate --size 2G --name lv00lclu cluvg00
/sbin/mkfs -t gfs2 -p lock_dlm -j 2 -t bldg1ux01clu:lclu /dev/cluvg00/lv00lclu 

mkdir /data/lclu/samba
mkdir /data/lclu/samba/ctdb
mkdir /data/lclu/samba/drivers
chmod 775 /data/lclu/samba/drivers
chown root:itstaff /data/lclu/samba/drivers

The "itstaff" group above are the people allowed to add drivers to the server.

I now edit /etc/sysconfig/ctdb on both nodes, the options I have are:


In /etc/sysconfig/samba I have (both nodes):
# Options to smbd
# Options to nmbd
# Options for winbindd

I prefer using port 445, which should be lighter weight than port 139 (with it's netbios wrapping).

Now I put into  /etc/ctdb/nodes the IP addresses of my private network (clustered samba will use these for internal comms) (both nodes):

Then in /etc/ctdb/public_addresses I put in the public IP that I want this service to run at, with mask , again on both nodes:

Then just my modified /etc/samba/smb.conf:

        workgroup = MYDOMAIN
        clustering = yes
        netbios name = bldg1clusmb
        max log size = 100
        preserve case = yes
        short preserve case = yes
        security = ADS
        realm = MYDOMAIN.LAN
        password server = dclan.lan
        encrypt passwords = yes
        load printers = yes
        local master = no
        client use spnego = yes
        log level = 1
        printing = cups
        printcap = cups
        cups options = "raw"
        use client driver = no
        printer admin = @itstaff
        map to guest = Bad User
        guest account = guest

        printable = yes
        path = /var/spool/samba
        browseable = no
        public = yes
        guest ok = yes
        writable = no
        default devmode = yes

        comment = Windows Printer Driver Download Area
        path = /data/lclu/samba/drivers
        browseable = no
        guest ok = yes
        read only = yes
        write list = @itstaff
        force group = +itstaff
        force create mode = 0775
        force directory mode = 0775
; Local disk configurations
        guest ok = no                                                                                                                                 
        writeable = yes                                                                                                                               
        path = /data/projects                                                                                                                         
        force create mode = 0664
        force directory mode = 0775
        guest ok = no                                                                                                                                 
        writeable = yes                                                                                                                               
        path = /data/home                                   

The first thing to note is I don't have a homes share. This is because I use the automounter. Any homedirs I mount from the cluster will go via NFS (as they refer to a service IP (by the name bldg1clunfshome) ), samba mounts via NFS tend not to work very well (due to locking issues) and will be slower. So I have created a "user" share that people can get straight to their homedirs via.

I add the public addresses to DNS and hosts.   bldg1clusmbA bldg1clusmbA.lan   bldg1clusmbB bldg1clusmbB.lan

BUT this time we want to add to DNS bldg1clusmb that will point to two A records, one for each service IP address that I'm using for samba e.g

# nslookup bldg1clusmb

Name:   bldg1clusmb.lan
Name:   bldg1clusmb.lan

I also have an "netbios name = bldg1clusmb" parameter, cause as I'm using AD I need to join the Samba to AD with the name the clients will refer to it as. But you'll need to start it before joining. 

Stop any samba's running on the nodes and chkconfig them off.

/etc/init.d/smb stop
/sbin/chkconfig smb off

And on both nodes start the ctdb service and chkconfig on:

/etc/init.d/ctdb start
/sbin/chkconfig ctdb on

You can check the status of the ctdb with "ctdb status". It will take a little while to settle but eventually it should return:

Number of nodes:2
pnn:0    OK (THIS NODE)
pnn:1    OK
hash:0 lmaster:0
hash:1 lmaster:1
Recovery mode:NORMAL (0)
Recovery master:1

Then you can join to AD or what ever and connect from Windows clients using \\bldg1clusmb and DNS should now round-robin between the two nodes from various clients. 

There is a bit of an issue with printing. Sadly the registry information about printers doesn't get copied between the nodes as yet. My hack around this is to stop the ctdb on one of the nodes. Then install the printers or printer drivers on the server that's up (just using the cluster name \\bldg1clusmb). Then when finished, copy the files ntdrivers.tdb, ntforms.tdb, ntprinters.tdb and the printing directory (and contents) all in /var/lib/samba to the other (down) node's /var/lib/samba. Then restart ctdb on the down node. As they share the driver directory this should now allow both nodes to perform click to print auto client driver installs. Just remember to do this procedure every time you make printer driver changes (or any default settings on these printers for windows). A bit of a hassle, but seems to work.

No comments:

Post a Comment