Posts Tagged ‘CentOS’

CentOS 6 with SCST

I keep meaning to write down my SCST installs so the other network engineers here at work can reference how I did it all. This documentation is also for me too (well, primarily for me actually!)

Here it goes…

1.       install CentOS:

a.       While on Hostname screen, select “configure Network” and check “connect automatically” on your eth0 device so it will install & configure basic networking alongside NetworkManager.

b.      Select the “Software Development Workstation” option then unselect groups virtualization, virtualization-client and virtualization-platform or let the installer finish then use “yum groupremove virtualization virtualization-client virtualization-platform” once you’ve booted into the OS the first time. By selecting the “Software Development Workstation” as your install type, it will pretty much install every package you’ll need to do the kernel compile.

2.       Now do a yum update and reboot the box so you’re using the new kernel.

3.       Install some remaining dependencies for this full kernel build:

a.       yum install asciidoc

b.      yum install newt-devel

4.       Commence kernel build prep.

a.       Find out which kernel you’re currently using w/ “uname -r” and write it down.

b.      Create user for building kernel & SCST by running “useradd mockbuild” go ahead and change from user root to that user “su – mockbuild” for the next step (no, mockbuild doesn’t need a password so don’t give it one!)

c.       While as the mockbuild user, execute “mkdir -p ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}” which creates the directory structures for the build and finally “echo ‘%_topdir %(echo $HOME)/rpmbuild’ > ~/.rpmmacros” which creates your .rpmmacros file w/ the build directory location. (NOTE: spacing is very important on this command. Just copy & paste to be sure you typed it correctly.)

d.      Again, while as the mockbuild user, execute “rpm -i http://vault.centos.org/6.4/updates/Source/SPackages/kernel-2.6.32-358.14.1.el6.src.rpm 2>&1 | grep -v mock” which will install the current kernel source and put the files in your build tree folders (while also redirecting stderr to stdout while grepping it.)

e.      While in your homedir and as the mockbuild user, execute “rpmbuild -bp –target=$(uname -m) ./rpmbuild/SPECS/kernel.spec” so it will uncompress the kernel sources and move them to your ./rpmbuild/BUILD folder. (NOTE: While it’s running, open up another console and as root run “rngd -r /dev/urandom” as the certificates will drain out the random number generator and kind of hang the rpmbuild.)

5.       Download SCST from svn & patch kernel in BUILD directory.

a.       We need to grab a copy of SCST using svn. There’s a number of branches but we want the latest stable. So run “svn co https://svn.code.sf.net/p/scst/svn/branches/2.2.x scst” while the mockbuild user and in /home/mockbuild directory. This will
be the latest stable version of SCST which is 2.2.1.

b.      Now that our user mockbuild has a copy of scst in his home directory, go ahead and cd into the unpacked kernel so we can patch it. Should be “cd ~/rpmbuild/BUILD/kernel-2.6.32-358.14.1.el6/linux-2.6.32-358.14.1.el6.x86_64/”.

c.       We need to patch for SCST’s fifo vs lifo: “patch -p1 < ~/scst/scst/kernel/rhel/scst_exec_req_fifo-2.6.32.patch”

d.      We need to patch for iscsi-scst’s put_page_callback: “patch -p1 < ~/scst/iscsi-scst/kernel/patches/rhel/put_page_callback-2.6.32-220.patch”

6.       Time to compile the Linux kernel & install it.

a.       Check your .config files (defualt one inside your ~rpmbuild/BUILD/kernel-*/linux-*/.config to your current kernel config in /boot/config-2.6.32-*) They shoud pretty much match. This means you can skip “make oldconfig” which would prompt you for stuff missing from your provided .config file with what this kernel you’re about to compile has as of now (meaning, if you’re compiling a newer kernel, your .config file is missing some stuff so it will prompt you for defaults and update your older .config file!)

b.      Change the kernel name by editing the Makefile. We need to separate this kernel from the others by editing the EXTRAVERSION varible. Change it to “=-scst” version versus being just empty or blank.

c.       Now run “make menuconfig” and change the following settings for SCST:


i.  Select “Network support”->”Networking options” make sure “TCP/IP networking” is checked (should be) and check “TCP/IP zero-copy transfer completion notification” underneath so SCST will be notified when a transfer of data across the nic has finished. This way, it can hurry up & send the next block. Should speed things up.


ii.   Next, verify that “Device Drivers”->”SCSI device support”->”SCSI disk support” is already checked. Again, for RHEL 6.4 it should be.


iii.  Verify that “Enable the block layer”->”IO Schedulers”->”CFQ I/O scheduler” and “Default I/O scheduler” set to CFQ are both set. Again, they should be.


iv.  Lastly, set “Processor type an features”->”Preemption Model” is set to “No Forced Preemption” versus Desktop mode.


v.  Once done, save your config so we can compile this kernel.d.      Once that’s done, we can compile it. Type “time make -j8 all” as the mockbuild user. This makes the kernel & modules while timing it and using all of your CPU’s cores (for me, I’ve got 8 cores so -j8!)

e.      Now we need to install our new kernel. Do a “make modules_install” as root which will copy the modules over to /lib/modules/<kernel name> directory (you did call your kernel something different by setting the EXTRAVERSION variable right?) After it’s done, do a “make install” as root as well which will update the /boot directory & grub. Now update grub.conf so it will default to our new kernel upon boot up.

f.       Once that’s done, copy over the current .config file in the /home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-358.14.1.el6/linux-2.6.32-358.14.1.el6.x86_64 directory to the /boot directory and name it such that it matches your other config files in that directory (such as config-2.6.32-scst.)

7.       Reboot your system so we can use the new kernel and compile scst off the new kernel source files.

8.       Now we’ll compile SCST.

a.   Go ahead and change the various Makefiles at once to release version by issuing “make debug2release” in the main scst directory. This will remove the debugging stuff. Do this as root. Also, since CentOS 6 now uses sysfs and not proc, we don’t have to do a “make enable_proc” like for CentOS 5.

b.      Now go ahead and issue “make scst scst_install” as root in the mockbuild directory’s scst folder.

c.       Issue “make iscsi iscsi_install” as root in your current folder.

d.      Finally, issue “make scstadm scstadm_install” as well. This finalizes the compiling part.

9.       Configure & start SCST.

a.       Need to go ahead and tell our system to start it up automagically on bootup so execute “/usr/lib/lsb/install_initd scst” as root user.

b.      Go ahead and either reboot to test it or “service scst start” to fire it up. (NOTE: We haven’t done a config file yet so it’s running w/ nothing up!)

c.       Next, we need a simple config. Since I’m using actual RAID volumes as disks in SCST, I’ve gone w/ BLOCKIO devices and list them as /dev/sdX devices. You may want to use FILEIO devices instead or even a different device handler all together (scst_disk instead of scst_vdisk.) Here’s a couple of different examples:


i.      dev_disk example which uses scst_disk:

                                                HANDLER dev_disk {

                                                                                DEVICE 3:0:0:0

                                                }

                                                TARGET_DRIVER iscsi {

                                                                                enabled 1

                                                                                TARGET iqn.2013-07.com.example.san1:iscsi-tgt1 {

                                                                                                                enabled 1

                                                                                                                LUN 0 3:0:0:0

                                                                                }

                                                }


ii.     vdisk_blockio example which uses scst_vdisk:

                                                HANDLER vdisk_blockio {

                                                                                DEVICE disk01 {

                                                                                                                filename /dev/sdb1

                                                                                }

                                                                                DEVICE disk02 {

                                                                                                                filename /dev/sdc1

                                                                                }

                                                }

                                                TARGET_DRIVER iscsi {

                                                                                enabled 1

                                                                                TARGET iqn.2012-09.com.example.san1:iscsi-tgt1 {

                                                                                                                enabled 1

                                                                                                                LUN 0 disk01

                                                                                                                LUN 1 disk02

                                                                                }

                                                }


iii.     vdisk_fileio example which also uses scst_vdisk:

HANDLER vdisk_fileio {

                                                DEVICE mydisk01 {

                                                                filename /home/testdisk1

                                                                nv_cache 1

                                                }

}

TARGET_DRIVER iscsi {

                                enabled 1

                                TARGET iqn.2013-07.net.spiderbird.san2:iscsi-tgt1 {

                                                                enabled 1

                                                                LUN 0 mydisk01

                                }

}

10.   Finalizing your install:

a.       Now that we’re done, it’s best to configure yum not to update your kernel w/ a newer one which would break all of your hard work. Modify your /etc/yum.conf file and add “exclude=kernel*” to it w/o the double quotes. This will help keep your system somewhat up-to-date w/o breaking SCST & your custom kernel.

11.   Troubleshooting:

a.       Keep in mind, if you’re going to use multiple SCST SANs in your backend, you need to make sure all of your exposed LUNs have different t10_dev_id values else your host will get confused and assume it’s seeing different paths to similar items. Also, using setup_id is supposed to help prevent this but evidently doesn’t work in version 2.2.x. You can skate around this issue by using different names for your defined DEVICEs (instead of using disk01 try using SAN1disk01.)

b.      Pass through devices work as expected except for when that pass through device is a simple SATA drive. Your ESXi 5+ server still won’t put a VMFS partition on there because it isn’t reliable storage. Best to stick w/ RAID volumes in pass through mode instead.

c.       I forget C….

Here’s a couple of links that I used during this write up:

Kernel compile links:

http://wiki.centos.org/HowTos/I_need_the_Kernel_Source

http://wiki.centos.org/HowTos/Custom_Kernel

SCST info regarding speed, settings & setups:

http://scst.sourceforge.net/max_outstanding_r2t.txt

http://scst.sourceforge.net/iscsi-scst-howto.txt

http://scst.sourceforge.net/vl_res.txt

http://sourceforge.net/p/scst/svn/4920/tree/branches/2.2.x/ (SCST SVN)

CentOS FAQ regarding install groups:

http://wiki.centos.org/FAQ/CentOS6#head-ef37a027209dcdba0f8f00cf2696721b774f74bf

A bunch of different write-ups for various Linux installations:

http://www.howtoforge.com/how-to-create-a-fiber-channel-san-using-scst-with-qlogic-hba-on-linux-debian-6  Debian…

http://aliuhui.blogspot.com/2012/04/setup-iscsi-target-with-iscsi-scst-20.html  Ubuntu …

http://jaysoncena.wordpress.com/2013/03/17/26/  generic..

http://temporalsolutions.com/linux-ha/scst-setup/ Ubuntu..

http://wiki.alpinelinux.org/wiki/High_performance_SCST_iSCSI_Target_on_Linux_software_Raid  Alpine Linux

http://monklinux.blogspot.com/2012/02/scst-configuration-how-to-using-gentoo.html Gentoo

http://www.bauer-power.net/2012/03/how-to-install-scst-on-ubuntu.html Ubuntu..

http://www.kcsts.co.uk/blog/scst-iscsi Ubuntu again…

http://www.lessfs.com/wordpress/?p=850

http://www.lessfs.com/wordpress/?page_id=577

http://marcitland.blogspot.com/2011/03/accelerating-vdi-using-scst-and-ssds.html Gentoo

Here’s a number of links regarding iSCSI settings you can tweak and what they mean. These are primarily geared towards using your SCST installation with ESXi and MPIO:

https://communities.vmware.com/docs/DOC-15437

http://deinoscloud.wordpress.com/2010/08/19/vsphere-4-1-iscsi-advanced-settings-and-their-meanings/

http://forum.open-e.com/archive/index.php/t-2537.html (Tuning iSCSI settings)

http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vsphere.html Best link I’ve found yet of tweaking your ESX iSCSI settings.

http://www.gatt-it.co.uk/1/post/2012/11/how-to-set-the-round-robin-iops-setting.html

http://forum.open-e.com/showthread.php?1542-VMWare-vSphere-Recommended-settings

https://forums.openfiler.com/index.php?/topic/4681-howto-definitive-esxi-41-configuration-guide/

 

 

 

 

Properly Partitioning a HW RAID Volume over 2TB in size

Like most geeks I’ve got lots of files and a need for spacious network shares. However, as soon as you go over a 2TB partition, you’ll encounter problems w/ how to correctly partition a drive or RAID volume using the newer 4096 sector spec called simply Advanced Format. I initially stumbled upon a series of articles written by Roderick W. Smith over on IBM’s DeveloperWorks when searching how to properly check partition alignment under Linux for SSDs but quickly had an “Uh Oh” moment when I realized SSDs weren’t my only problem and I had in fact been flying blind on larger than 2TB partitions under Linux. Thankfully I didn’t have systems in production where this was a problem but I was literally building a system at work where this could bite me in the rear!

To bottom line it for you, if you’re using software RAID or a true HW RAID setup, you need to stripe your file system across the partition in accordance to the number of discs involved, type of RAID and the stripe size in KBs. My example below is strictly regarding Areca ARC RAID cards and might not be applicable to your setup.

I’m using an ARC-1680ix-24 w/ 4GB of ram on board. I’ve got 24 – 500GB HDs with one volume using all HDs in a RAID 6 raidset. So, for me the math is such when using EXT4 filesystem:

chunk size = 128KB (For Areca ARC-1680s it's your Stripe size.)
block size = 4KB (My desired partition is over 2TB. 6.5TB to be exact.)
stride = chunk / block 
       = 128KB / 4KB 
       = 32
Stripe-width = stride * (( # disks in RAID) - # of RAID parity disks)
             = 32 * ((using all 24 disks) - RAID 6 uses 2 parity disks)
             = 32 * (24 - 2)
             = 32 * 22
             = 704

so in other words, once I’ve created the basic partition on my device and I’m ready to format it as EXT4 I’ll end up using:

mkfs.ext4 -v -m .1 -b 4096 -E stride=32,stripe-width=704 /dev/sdd1

I’m not going to go into every detail of the line so please read RAID Setup over on Kernel.org for the skinny.

 
So from Soup to Nuts:

    Use parted to create the partition table on my new volume & create a massive single partition at the right offset:
parted /dev/sdd
    Once in the parted tool:
mklabel gpt
unit s
mkpart primary 2048s 100%
name 1 BFS
quit

 

    Next, you’ll need to create an EXT4 filesystem inside your newly created partition from above:
mkfs.ext4 -v -m .1 -b 4096 -E stride=32,stripe-width=704 /dev/sdd1
    Now you’ll want to add it to your fstab file so let’s grab the UUID from blkid next:
blkid -o list
    And edit your /etc/fstab file so it will automount it at boot time
.
UUID=8e0a7d10-blah-blah-tomatoes-are-yummy-b4a0f6a13c15 /bfs ext4 defaults  1 2
.
    Finally, create the mount point and mount it.
sudo mkdir /bfs
mount /bfs

Here’s a couple of links I found note worthy while going down this rabbit hole:
https://raid.wiki.kernel.org/index.php/RAID_setup#Calculation (shows the actual formula shown above & values)
http://insights.oetiker.ch/linux/raidoptimization.html (a great read)
http://ubuntuforums.org/showthread.php?t=1715375  (Rod weighs in with helpful advice in the Ubuntu forums)
http://www.gnu.org/software/parted/manual/parted.html  (parted’s man page over on GNU.ORG)
http://en.wikipedia.org/wiki/Ext4
http://lwn.net/Articles/377897/  (talks about 4KB sector size disks and let the panic ensue!)
http://whattheit.wordpress.com/2011/08/23/linux-aligning-partitions-to-a-hardware-raid-stripe/  (lots of theory but looks incomplete)
Last 4 links are from Rod Smith:
http://www.rodsbooks.com/gdisk/advice.html  (using gdisk but applicable to parted)
http://www.rodsbooks.com/gdisk/index.html  (main gdisk site!)
http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/  (talks about the severe performance effects if you gloss over this stuff!)
http://www.ibm.com/developerworks/linux/library/l-gpt/  (good overview of GPT & understanding why they’re moving away from MBR)

(Personal Note: By finally posting this up on my blog I can close 10 tabs I’ve had open since middle of 2011!)

Sendmail’s tweakable settings in the sendmail.mc file

Recently had a question at work about the maximum number of recipients Sendmail would allow thru our in-house server. I wasn’t sure but knew I hadn’t personally configured a hard limit. Did some quick searching and stumbled upon confMAX_RCPT_PER_MESSAGE as the answer. I haven’t set it to anything in our config but didn’t know what the default limit was in Sendmail so I did some digging. Stumbled upon the following links of value (to me at least!)

http://www.sendmail.com/sm/open_source/docs/m4/tweaking_config.html

Answer: infinite number. Sendmail doesn’t have a limit (now it does!)

Also stumbled upon Centos.org’s version 5 Deployment Guide on running the m4 macro command to build a new Sendmail.cf file.

http://www.centos.org/docs/5/html/5.1/Deployment_Guide/s3-email-mta-sendmail-changes.html

 

Linux bonding: How to maximize your throughput by combining nics.

I’m currently working on building my own iSCSI SAN w/ SCST & a modified Linux kernel. Things are going pretty well but I needed to also bond out the 5 Gb nics so my VMHosts can maximize their connections. In Linux, you bond out an interface by creating a new network interface and telling the physical nics to register w/ that virtual network interface. There’s also 7 different types of bonding modes but the most common one is mode 6 or balance-ALB (active load balancing.)

If you’re using CentOS or RHEL like I am you cd to /etc/sysconfig/network-scripts/ and vi ifcfg-bond0.

Once you’ve got vi open, you dump the following in (this is an example only. Please change your IPADDR, NETMASK and NETWORK parameters as needed.)

DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
NETWORK=10.0.0.0
NETMASK=255.255.0.0
IPADDR=10.0.0.100
USERCTL=no

From there, you then modify your individual eth# interface config files so they point to the bonded device. As an example, here’s my /etc/sysconfig/network-scripts/ifcfg-eth0 interface file.

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

And oh ya, don’t forget to modify your modprobe.conf file to specify the loading of the bond kernel module and what options it will needed @ run-time.

alias bond0 bonding 
options bond0 mode=6 miimon=100

Couple of useful links. There’s tons of good info on bonding w/ Linux. Keep in mind, some distros don’t want you to modify the individual interface config files so use their administrator control panel first if possible.

http://www.how2centos.com/how-to-set-up-network-bonding-with-centos-5x/
http://www.howtoforge.com/network_card_bonding_centos

Installing nVidia driver under CentOS & Red Hat 6.0

I’m a real big fan of nVidia’s Linux drivers for their GPUs and have installed them numerous times with CentOS 5. I just recently installed them under Red hat 6 (RHEL 6) and happened to notice a warning by the nVidia driver that Nouveau was already installed and needed to be removed first. Further investigation reveals that Fedora & the latest version of RHEL 6 are including Nouveau at the initial ramdisk so not only do you need to create your own modprobe.conf blacklist file but you also need to add a line on the end of your grub.conf file to tell the ramdisk not to load Nouveau at boot up.

First, creating a modprobe blacklist file to prevent the kernel from loading the Nouveau kernel module. You could easily add blacklist nouveau to any ole modprobe.conf file but you never know when your linux distribution may overwrite your modified file w/ the next update. You’re better off to just create your own modprobe blacklist file and dropping it inside /etc/modprobe.d/ folder. I went ahead and followed some directions I stumbled upon on the interwebs and created a file called /etc/modprobe.d/disable-nouveau.conf.

I then added the following to that file…

blacklist nouveau options nouveau modeset=0

Once that was done, I then needed to modify my /etc/grub.conffile so the initial ramdisk would stop loading Nouveau too. To do this I added rdblacklist=nouveauto my kernel vmlinuz line like such…

.
.
.
kernel /vmlinuz-2.6.32-71.14.1.el6.x86_64 ro root=UUID=209502fb-f4f0-4755-a275-de807916fb76 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb rdblacklist=nouveau
.
.
.
.

Oh ya, I always remove the “quiet” at the end of the line too. Something about watching all of the modules load…

Here’s a couple links as usual:

http://us.download.nvidia.com/XFree86/Linux-x86_64/260.19.36/README/commonproblems.html
http://www.if-not-true-then-false.com/2009/howto-install-nvidia-driver-on-fedora-and-just-disable-the-nouveau-driver/

Custom Vanilla kernel for CentOS

I’m in the process of building up a SAN from scratch using SCST & CentOS. I’ve played around w/ kernels in the past but hadn’t done it in a while. When downloading 2.6.36.2 from Kernel.org recently I couldn’t get a bootable kernel no matter what I did w/ CentOS (and yes, I did load the CentOS .config file into menuconfig before compiling my kernel.)

  Come to find out, because CentOS is using a modified 2.6.18 kernel that little .config file is very out dated. As of kernel 2.6.31 a new kernel parameter is available called “enable deprecated sysfs features to support old userspace tools” which is under the General setup area. Once I enabled that option, I was good to go. It has something to do with why my SAS RAID card wasn’t getting picked up upon boot up. Everything else after that was a piece of cake!

You could have also just added this to your .config file. (NOTE: Hats off to Vanecka!)
CONFIG_SYSFS_DEPRECATED_V2=y

Couple Kernel compile links for CentOS 5:

http://wiki.centos.org/HowTos/Custom_Kernel  (official CentOS wiki kernel build page)
http://www.howtoforge.com/kernel_compilation_centos_p2  (HowtoForge article)
http://kerneltrap.org/node/1783 (Old kernel trap article which mentions the use of -j8 for make parameters so you’ll use all of your cores on compiles! I did a ‘make -j8 all’ and finished my kernel compile in under 5 minutes on a Xenon E5620 CPU!)
http://www.linuxfromscratch.org/lfs/view/6.4/chapter08/kernel.html (LFS link on kernel compile)

quick tips!

 

Moving lots of data:I do all sorts of networking related stuff and one of the few things I do often are server migrations. So eventually, there comes a time for me to copy a large amount of data from one server to another while maintaining the file permissions and structure. That’s where tar & ssh come into play. If I’m moving a users’ home directory from one server to another I usually use tar & ssh together via a pipe and let it copy off the files perfectly.

tar zcvf - jsmith | ssh "destinations IP" "cd /home/; tar zpxvf -"

Within seconds of the command starting off, you’ll be asked for the password for your ssh session. Then, the data will get flung across the network to the destination folder. Works great!

Verifying Apache Configuration:When adding additional Virtual Hosts to your Apache config file it’s nice to make sure they’ve been picked up correctly (or! that you’ve modified the right httpd.conf file!) A simple way of letting Apache tell you what it’s going to use is via the -t option.

httpd -t -D DUMP_VHOSTS

It parses the running configuration and dumps the virtual hosts a nice readable format. You can also use “DUMP_MODULES” to see a list of the modules it will use which can be useful for whether or not that database module is going to be loaded or not.

MySQL database dump:It never fails, when doing a migration you’re going to need to dump your databases from a MySQL DB into another DB on your newer system. If they’re the same version however you can run mysqldump w/ a couple command-line switches and then import them on the new server.

mysqldump -u root -p --databases "your DB here" > dbfilename.sql

After you’ve moved your database file to the new server you can then import it in via:

mysql -u root -p -D "your DB here" < /path/to/dbfilename.sql

Keep in mind, we’re assuming you’ve already created the database inside your new MySQL server and will assign it proper permissions so your end user can connect to it!

Just a couple quickies for today’s post.

As always, a couple useful links:

http://www.debuntu.org/how-to-create-a-mysql-database-and-set-privileges-to-a-user

http://www.webmasterworld.com/forum49/723.htm

http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html

http://dev.mysql.com/doc/refman/5.0/en/default-privileges.html

vsftpd, FTPES & CentOS 5.5 part 2

 

This is the second part of my FTPES, vsftpd & CentOS article. In the first part I walked you thru how I built a certificate, sent the CSR off to my CA and finally, modified the vsftpd.conf file. In this part I’ll show you how to test the service via command-line so you can actually see the certificate and how to punch a hole thru your FW because the data part of your FTP session is now encrypted.

First, let’s talk about verifying the configuration. One of the obvious things you can do is open up a command prompt session and attempt to log into your ftp server. Once connected, does it accept an anonymous connection? Was that what you wanted? Also, was the plain jane ftp command allowing you to log in or did you get a 331 error, “Non-anonymous sessions must use encryption.” If you’ve configured it correctly it shouldn’t allow you to login at all. So how do you test it using encryption? By using openssl w/ s_client of course! Use…

openssl s_client -starttls ftp -connect yourserver.example.com:21

This will allow you to not only log into your server using encryption via command-line but also verify that you’ve got the proper certificate & certificate chain installed. I personally tried connecting w/ out the starttls option but wasn’t successful (instead I get a “SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol” error message. I beleive this is because I’m using vsftpd in explicit mode not impicit mode.) The openssl s_client command can also be used for a number of other encrypted services for debugging certificates which is extremely helpful!

So you’ve verified that FTPES does work on the local LAN but doesn’t seem to work thru your FW. This is where passive ftp comes into play because your FTP traffic is now encrypted and your FW can’t do an inspection to determine which port is going to be used next and open it up for you ahead of time. Most FWs come with some sort of FTP inspect feature but we just killed it because that data is totally encrypted now and it’s unable to sniff the FTP traffic and sense which port needs to be opened. You won’t have to turn on passive FTP because it’s on by default with vsftpd. You should specify the port range and I also turned on pasv_address too:

pasv_enable=YES
pasv_addr_resolve=YES
pasv_address=yourserverhere.example.com
pasv_min_port=63000
pasv_max_port=65535

Once done, this will force your vsftpd server to use a port range of 63000 to 65535 for data connections. Alot of the commands I used above can be found w/in the man page for vsftpd.conf (along with their default values!)

Now you just need to open up that same range on your FW device (NOTE: Each FW is going to be different!) On my Cisco Adaptive Security appliance (ASA) I’m defining an object-group first for the port range like such:

object-group service ftp_passive tcp
  description ftp passive range
  port-object range 63000 65535

Then I use that port range in my outside ACL for inbound traffic like such:

access-list outside_access_in extended permit tcp any host YOUR_PUBLIC_IP object-group ftp_passive
.
.
.

Just be sure to use your public IP address for ‘YOUR_PUBLIC_IP’ in the above ACE. Now when it comes to ACL rules there’s tons of different ways to allow traffic so I won’t go into much more detail here other than how I punched a hole thru my FW to allow the encrypted vsftpd data channel traffic through.

That’s about it folks. Remember, if you’re having problems with your configuration, break it down into simple pieces and troubleshoot it that way versus trying to eat the entire problem at once!

-Q

Oh, almost forgot. Here’s a couple links that might be helpful!

http://en.wikipedia.org/wiki/Ftp
http://en.wikipedia.org/wiki/FTPES
http://it.toolbox.com/blogs/unix-sysadmin/troubleshooting-ssltls-mail-services-29266
http://www.wowtutorial.org/tutorial/26.html
http://vsftpd.beasts.org/
http://www.openssl.org/docs/apps/s_client.html
http://blogs.iis.net/robert_mcmurray/archive/2008/11/10/ftp-clients-part-2-explicit-ftps-versus-implicit-ftps.aspx

DNSBL, Sendmail and my Aunt…

So this past weekend I went to visit my Aunt & Uncle in Harrisonburg, VA. While there, my Aunt tells me she can’t send email thru my server which I was fairly certain was once again, “user error.” Come to find out. It was a misconfiguration on my part! Ah ha!

Long story short, I had reconfigured my email server about two months ago and added a DNSBL (zen.spamhaus.org) to my sendmail features and it was blocking her from relaying email thru my sendmail server. I simply had to add…

FEATURE(`delay_checks')

… to my sendmail.mc file and restart the service. (NOTE: On CentOS 5.0 if you restart the sendmail service it will check your sendmail.mc & sendmail.cf files and do a ‘make -C /etc/mail’ for you if the sendmail.mc is newer.)

Now, Sendmail will wait on the DNSBL and give the user a chance to authenticate first which will allow them to then bypass the DNSBL. Seems her residential comcast IP Address was added to zen.spamhaus.org which is commonly done to blocks of residential user IP address spaces to prevent spam.

vsftpd, FTPES & CentOS 5.5 part 1

Ask anyone that knows me and they’ll tell you I love Linux. In particular, I love Red Hat’s flavor of Linux. I’ve still got my first Linux book “The Complete Red Hat 5.2 Installation Guide” and while I don’t consult it first for all problems I do remember thumbing thru its pages when trying to configure those tulip.o drivers and how to change my X Windows config file settings (NOTE: I still to this day get tripped up on configuring X Windows!) So that having been said, it’s no wonder I’m using CentOS and have been for years since Red Hat has stopped releasing a free desktop variety for users to use (I know they’ve got Fedora but that product lines’ release cycle is too aggressive for my tastes.) 

  So that having been said, I love CentOS. I’ve been using it at home for a while now and whenever possible do my best to stick w/ the stock rpms so updates go smoothly. I’ve recently been trying to get vsftpd daemon to use encryption while still jailing my end users in their home directories and here’s what I found out. Vsftpd doesn’t natively do sftp but it does do ftpes pretty well. So I’m going to walk thru generating a private key, making a certificate signing request, adding the public key from your third party certificate authority (CA) to their certificate bundle to form a chained public key and finally, configuring vsftpd to do ftpes.

First, I logged into the box and grabbed superuser.

su -

Then changed directory to the location of the certificates.

cd /etc/pki/tls/certs/

Next, I ran openssl to generate my private key w/o a passphrase! (NOTE: If you decide to use a passphrase a number of services won’t auto start on a power failure which means you’ll need to run home, start the service manually again and enter in the passphrase so the service can use the certificate. Some services can be configured to manually put the passphrase in for you but that’s lots of effort.)

openssl genrsa -out myserver.mydomain.private.key 2048

Had I wanted to generate a private key w/ a passphrase it would have been…

openssl genrsa -des3 -out myserver.mydomain.private.key 2048

You’ll notice at the end of the command I put 2048. That stands for how many bits this certificate is generated with. A number of CAs will require now a days that you use a 2048 bit key versus the old standby of 1024 bit ( the CA I use GoDaddy is one of them.)

The very next thing we need to do is lock down your private key. This is used to encode encrypted communications to the end user who will now use your public key to decrypt the data. So let’s make it readable only by root.

chmod 600 myserver.mydomain.private.key

Once that’s done, it’s time to generate our certificate signing request (CSR) that needs to be passed to your CA so they can generate a public certificate for you.

openssl req -new -key myserver.mydomain.private.key -out myserver.csr

At this point you’ll be prompted for answers to a series of questions which will be placed in your CSR. The big thing you need to get right is your Common Name (CN) question. Make sure the CN is the server name that matches up w/ your DNS entry for this server. Folks are going to use DNS to access your server so the certificate needs to match up with what they are going to use to hit your server across the internet. It’s interesting to note, that w/ GoDaddy, pretty much all of the other fields you fill in get thrown out in the generated public certificate they send back to you so don’t get too freaked out with putting your location information in there. Again, this is with GoDaddy. Other CAs may publish that information w/ the public certificate.

After you’ve submitted your CSR to your CA. You’ll have to go thru the whole domain verification process which can be its’ own little tap dance. I know when our company had their domain names register w/ someone else besides GoDaddy it took 24 to 48 hours for whois verification and whatnot. Once we moved our domain names to GoDaddy the verification process took less that 5 minutes. Coincidence? Business strategy? I don’t know but I’m sure glad we moved our domains over to GoDaddy.

Okay, so you’ve finally gotten your certificate from your CA. Yay! Now comes the fun of lumping that certificate w/ the included CA bundle they sent you and configuring your vsftpd daemon.

First, concatenating the two files together into a combined public certificate file. Why? Cause vsftpd doesn’t have an option for the certificate chain file your CA gave you w/ your public certificate.

cat myserver.crt CA_bundle.crt > myserver.mydomain.public.key

This will put both files together into a combined file which will have all certificates. Now we can configure vsftpd and be done.

Open up the vsftpd.conf file w/ vi.

vi /etc/vsftpd/vsftpd.conf

When looking at your vsftpd.conf file I started at the beginning and slowly worked my way thru turning off features I knew I didn’t need. One of the first was anonymous_enabled. It defaults to enabled unless you uncomment the line and tell it NO. Most everything else I left as is. It’s interesting to note, that the SSL features you’re looking for in the config file aren’t present so you’ll need to add them manually to the file. Here’s what I’ve put in mine. Most of these commands by the way are covered in your man page for vsftpd.conf so take a look there for a more detailed explanation.

ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
rsa_cert_file=/etc/pki/tls/certs/myserver.mydomain.public.key
rsa_private_key_file=/etc/pki/tls/certs/myserver.mydomain.private.key

This essentially turns on SSL, turns off anonymous SSL, forces both data & logins to go over SSL and only allows TLS encryption. It also points to the location of your public & private keys.

This concluded part 1. Part 2 will talk about the implications of encrypting your FTP traffic and how it will mess w/ your FW rules at your border device.

-Q

Links I found useful while doing my install:
http://wiki.vpslink.com/Configuring_vsftpd_for_secure_connections_%28TLS/SSL/SFTP%29

http://en.wikipedia.org/wiki/FTPS

http://vsftpd.beasts.org/vsftpd_conf.html

Return top