Monday 2 January 2012

A RHEL 6/Centos 6 HA Cluster for LAN Services switching to EXT4 (Part 2)

Failover Multiple Samba Setup
As I wish to have two separate file systems exported via Samba, and by default they are configured to run on two different nodes (to provide basic load balancing), I will require multiple instances of Samba to be running. I can no longer use the built in clustering that Samba provides as this requires a clustered file system, that I'm eliminating from my setup. Also I need to isolate these two Samba instances from each other so they can coexist on the same node.

To do all this I have setup firstly two additional ext4 clvmd's, these will hold my Samba working files (for locking, spooling etc). I have called these cluvg00-lv00smbprj (demand mounted on /data/smbprj) and cluvg00-lv00smbhome (demand mounted on /data/smbhome). These are just created the same way as before.

I plan to use the "samba" resource agent. This will help me limit each Samba instance to only listen on it's service IP address.

I now need two Samba config files, one for my projects file sharing service and one for my home file sharing service. The main change to these config files is the addition of lines to direct all state to each instance of Samba's private directory.

So I have a smb.conf.home and a smb.conf.prj in /etc/samba (that must exist and be copied to both nodes).

Fragment of my smb.conf.prj:


netbios name = bldg1clusmbprj
private dir = /data/smbprj/var/lib/samba/private
lock directory = /data/smbprj/var/lib/samba
state directory = /data/smbprj/var/lib/samba
cache directory = /data/smbprj/var/lib/samba

Before doing anything else we need to populate the smbprj and smbhome filesystems. 
So once the fileystems are created temporarily mount the /data/smbhome and /data/smbprj. In each of these a very specific tree is required:

mkdir var etc
mkdir -p  etc/samba/drivers
cd etc/samba/drivers
mkdir COLOR W32ALPHA W32MIPS W32PPC W32X86 WIN40 x64
cd ../../..
cd var
mkdir lib log spool
mkdir lib/samba log/samba spool/samba 
chmod 1777 spool/samba

All directories should be owned by root with group root. The final chmod is required so users can have their print jobs spooled. All other directories are 755. 

The etc/samba/drivers directory is only required on the Samba instance you are planning to use for sharing printers (with auto driver install) to Windows systems (but will do no harm on the other Samba instance). You may want to set appropriate permissions on this to only allow your chosen group or users to upload drivers from a Windows system. See the standard Samba documentation for more information.


My cluster.conf fragments for this:


 <service autostart="0" domain="bldg1ux01Anfb" exclusive="0" name="nfsdprojects" nfslock="1" recovery="relocate">
                        <ip ref="10.1.10.29"/>
                        <fs ref="projectsfs">
                                <nfsexport ref="exportbldg1clunfsprojects">
                                        <nfsclient ref="nfsdprojects"/>                                                                                          
                                </nfsexport>
                        </fs>
                        <ip ref="10.1.10.32">
                                <fs ref="smbdprjfs"/>
                                <samba config_file="/etc/samba/smb.conf.prj" name="bldg1clusmbprj" smbd_options="-p 445 -l /data/smbprj/var/log/samba"/>
                        </ip>


                <service autostart="0" domain="bldg1ux01Bnfb" exclusive="0" name="nfsdhome" nfslock="1" recovery="relocate">
                        <ip ref="10.1.10.30"/>
                        <fs ref="homefs">
                                <nfsexport ref="exportbldg1clunfshome">
                                        <nfsclient ref="nfsdhome"/>
                                </nfsexport>
                        </fs>
                        <ip ref="10.1.10.33">
                                <fs ref="smbdhomefs"/>
                                <samba config_file="/etc/samba/smb.conf.home" name="bldg1clusmbhome" smbd_options="-p 445 -l /data/smbhome/var/log/samba"/>
                        </ip>
                </service>

A few notes on this. Firstly I have now included my Samba services in my nfsdhome and nfsdprojects cluster services. This is purely because I can't mount the filesystems I want to share on both nodes as it's ext4. I need all my services that depend on these file systems to start on the same node. The name I have for these cluster services could perhaps be more appropriate as it's no longer just nfsd.

I have however left separate IPs for SMB and NFS services, this is so I have in the future the option of going back to GFS2 and separating out these services again. Also despite the scoping of these "ip ref"'s you find that the one service's IP tends to "leak" into the config of the other service as generated by the resource script (in the above the NFS service IP of 10.1.10.30 will appear in the cluster generated config for samba under /etc/cluster/samba as well as the real one of 10.1.10.33) . This isn't really a problem so long as you don't try to use the wrong IP for the wrong service on the client machines (this will avoid future problems especially if Kerberos is involved on Samba or if you separate out services again).

The samba resource line has a few features. Firstly the "name" entry will be used in the generated samba config as the "netbios name" (though I put this in my source config too for clarity). The "-l" flag in the "smb_options" is used to direct all logging to my individual samba working directories (didn't see a good way of doing this in the config file), this is simply if two samba's are on the same node their logging won't trample on each other. I prefer to use "-p 445" so Samba listens on port 445 rather than 139. This simply means the client machines use direct SMB over TCP/IP rather than NetBIOS over TCP/IP, which is supposed to be cleaner and faster but there's probably not much in it.Again see the Samba documentation on this.

I'd recommend not having an /etc/samba/smb.conf so that no Samba can start apart from the cluster's ones. Also ensure that the "ctdb" and "smb" services are chkconfig'd off.

Once both Samba's are up and running, and you need to access the different samba's for different control commands you'll need to include the config file on their command lines. And these commands need to be run on the node that has that samba currently running on it. For example my Samba's are AD (Active Directory) joined. So the join command will be something like for my projects samba:

net ads join -U administrator -s /etc/samba/smb.conf.prj

or to get status

smbstatus -s /etc/samba/smb.conf.prj 

Before I used to want my http service to access things in my /data/projects directory, I no longer have this requirement. But if I did, it's simply a mater of moving the http service into the nfsdproject service like we have with Samba. Like this:

<service autostart="0" domain="bldg1ux01Anfb" exclusive="0" name="nfsdprojects" nfslock="1" recovery="relocate">                                
                        <ip ref="10.1.10.29"/>
                        <fs ref="projectsfs">
                                <nfsexport ref="exportbldg1clunfsprojects">
                                        <nfsclient ref="nfsdprojects"/>
                                </nfsexport>
                        </fs>
                        <ip ref="10.1.10.32">
                                <fs ref="smbdprjfs"/>
                                <samba config_file="/etc/samba/smb.conf.prj" name="bldg1clusmbprj" smbd_options="-p 445 -l /data/smbprj/var/log/samba"/>
                        </ip>
                        <ip ref="10.1.10.28">
                                <fs ref="httpdfs"/>
                                <apache config_file="conf/httpd.conf" name="httpd" server_root="/data/httpd/etc/httpd" shutdown_wait="10"/>
                        </ip>
                </service>

I'll put all my config files in the last section of this blog.