Geek projects, tidbits and things I'm working on at the moment…

Like most geeks I’ve got lots of files and a need for spacious network shares. However, as soon as you go over a 2TB partition, you’ll encounter problems w/ how to correctly partition a drive or RAID volume using the newer 4096 sector spec called simply Advanced Format. I initially stumbled upon a series of articles written by Roderick W. Smith over on IBM’s DeveloperWorks when searching how to properly check partition alignment under Linux for SSDs but quickly had an “Uh Oh” moment when I realized SSDs weren’t my only problem and I had in fact been flying blind on larger than 2TB partitions under Linux. Thankfully I didn’t have systems in production where this was a problem but I was literally building a system at work where this could bite me in the rear!

To bottom line it for you, if you’re using software RAID or a true HW RAID setup, you need to stripe your file system across the partition in accordance to the number of discs involved, type of RAID and the stripe size in KBs. My example below is strictly regarding Areca ARC RAID cards and might not be applicable to your setup.

I’m using an ARC-1680ix-24 w/ 4GB of ram on board. I’ve got 24 – 500GB HDs with one volume using all HDs in a RAID 6 raidset. So, for me the math is such when using EXT4 filesystem:

chunk size = 128KB (For Areca ARC-1680s it's your Stripe size.)
block size = 4KB (My desired partition is over 2TB. 6.5TB to be exact.)
stride = chunk / block 
       = 128KB / 4KB 
       = 32
Stripe-width = stride * (( # disks in RAID) - # of RAID parity disks)
             = 32 * ((using all 24 disks) - RAID 6 uses 2 parity disks)
             = 32 * (24 - 2)
             = 32 * 22
             = 704

so in other words, once I’ve created the basic partition on my device and I’m ready to format it as EXT4 I’ll end up using:

mkfs.ext4 -v -m .1 -b 4096 -E stride=32,stripe-width=704 /dev/sdd1

I’m not going to go into every detail of the line so please read RAID Setup over on Kernel.org for the skinny.

 
So from Soup to Nuts:

Use parted to create the partition table on my new volume & create a massive single partition at the right offset:

parted /dev/sdd

Once in the parted tool:

mklabel gpt
unit s
mkpart primary 2048s 100%
name 1 BFS
quit

Next, you’ll need to create an EXT4 filesystem inside your newly created partition from above:

mkfs.ext4 -v -m .1 -b 4096 -E stride=32,stripe-width=704 /dev/sdd1

Now you’ll want to add it to your fstab file so let’s grab the UUID from blkid next:

blkid -o list

And edit your /etc/fstab file so it will automount it at boot time

.
UUID=8e0a7d10-blah-blah-tomatoes-are-yummy-b4a0f6a13c15 /bfs ext4 defaults  1 2
.

Finally, create the mount point and mount it.

sudo mkdir /bfs
mount /bfs

Here’s a couple of links I found note worthy while going down this rabbit hole:
https://raid.wiki.kernel.org/index.php/RAID_setup#Calculation (shows the actual formula shown above & values)
http://insights.oetiker.ch/linux/raidoptimization.html (a great read)
http://ubuntuforums.org/showthread.php?t=1715375  (Rod weighs in with helpful advice in the Ubuntu forums)
http://www.gnu.org/software/parted/manual/parted.html  (parted’s man page over on GNU.ORG)
http://en.wikipedia.org/wiki/Ext4
http://lwn.net/Articles/377897/  (talks about 4KB sector size disks and let the panic ensue!)
http://whattheit.wordpress.com/2011/08/23/linux-aligning-partitions-to-a-hardware-raid-stripe/  (lots of theory but looks incomplete)
Last 4 links are from Rod Smith:
http://www.rodsbooks.com/gdisk/advice.html  (using gdisk but applicable to parted)
http://www.rodsbooks.com/gdisk/index.html  (main gdisk site!)
http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/  (talks about the severe performance effects if you gloss over this stuff!)
http://www.ibm.com/developerworks/linux/library/l-gpt/  (good overview of GPT & understanding why they’re moving away from MBR)

(Personal Note: By finally posting this up on my blog I can close 10 tabs I’ve had open since middle of 2011!)

Recently had a question at work about the maximum number of recipients Sendmail would allow thru our in-house server. I wasn’t sure but knew I hadn’t personally configured a hard limit. Did some quick searching and stumbled upon confMAX_RCPT_PER_MESSAGE as the answer. I haven’t set it to anything in our config but didn’t know what the default limit was in Sendmail so I did some digging. Stumbled upon the following links of value (to me at least!)

http://www.sendmail.com/sm/open_source/docs/m4/tweaking_config.html

Answer: infinite number. Sendmail doesn’t have a limit (now it does!)

Also stumbled upon Centos.org’s version 5 Deployment Guide on running the m4 macro command to build a new Sendmail.cf file.

http://www.centos.org/docs/5/html/5.1/Deployment_Guide/s3-email-mta-sendmail-changes.html

I’m currently working on building my own iSCSI SAN w/ SCST & a modified Linux kernel. Things are going pretty well but I needed to also bond out the 5 Gb nics so my VMHosts can maximize their connections. In Linux, you bond out an interface by creating a new network interface and telling the physical nics to register w/ that virtual network interface. There’s also 7 different types of bonding modes but the most common one is mode 6 or balance-ALB (active load balancing.)

If you’re using CentOS or RHEL like I am you cd to /etc/sysconfig/network-scripts/ and vi ifcfg-bond0.

Once you’ve got vi open, you dump the following in (this is an example only. Please change your IPADDR, NETMASK and NETWORK parameters as needed.)

DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
NETWORK=10.0.0.0
NETMASK=255.255.0.0
IPADDR=10.0.0.100
USERCTL=no

From there, you then modify your individual eth# interface config files so they point to the bonded device. As an example, here’s my /etc/sysconfig/network-scripts/ifcfg-eth0 interface file.

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

And oh ya, don’t forget to modify your modprobe.conf file to specify the loading of the bond kernel module and what options it will needed @ run-time.

alias bond0 bonding 
options bond0 mode=6 miimon=100

Couple of useful links. There’s tons of good info on bonding w/ Linux. Keep in mind, some distros don’t want you to modify the individual interface config files so use their administrator control panel first if possible.

http://www.how2centos.com/how-to-set-up-network-bonding-with-centos-5x/
http://www.howtoforge.com/network_card_bonding_centos

I’m a real big fan of nVidia’s Linux drivers for their GPUs and have installed them numerous times with CentOS 5. I just recently installed them under Red hat 6 (RHEL 6) and happened to notice a warning by the nVidia driver that Nouveau was already installed and needed to be removed first. Further investigation reveals that Fedora & the latest version of RHEL 6 are including Nouveau at the initial ramdisk so not only do you need to create your own modprobe.conf blacklist file but you also need to add a line on the end of your grub.conf file to tell the ramdisk not to load Nouveau at boot up.

First, creating a modprobe blacklist file to prevent the kernel from loading the Nouveau kernel module. You could easily add blacklist nouveau to any ole modprobe.conf file but you never know when your linux distribution may overwrite your modified file w/ the next update. You’re better off to just create your own modprobe blacklist file and dropping it inside /etc/modprobe.d/ folder. I went ahead and followed some directions I stumbled upon on the interwebs and created a file called /etc/modprobe.d/disable-nouveau.conf.

I then added the following to that file…

blacklist nouveau options nouveau modeset=0

Once that was done, I then needed to modify my /etc/grub.conffile so the initial ramdisk would stop loading Nouveau too. To do this I added rdblacklist=nouveauto my kernel vmlinuz line like such…

.
.
.
kernel /vmlinuz-2.6.32-71.14.1.el6.x86_64 ro root=UUID=209502fb-f4f0-4755-a275-de807916fb76 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb rdblacklist=nouveau
.
.
.
.

Oh ya, I always remove the “quiet” at the end of the line too. Something about watching all of the modules load…

Here’s a couple links as usual:

http://us.download.nvidia.com/XFree86/Linux-x86_64/260.19.36/README/commonproblems.html
http://www.if-not-true-then-false.com/2009/howto-install-nvidia-driver-on-fedora-and-just-disable-the-nouveau-driver/

I’m in the process of building up a SAN from scratch using SCST & CentOS. I’ve played around w/ kernels in the past but hadn’t done it in a while. When downloading 2.6.36.2 from Kernel.org recently I couldn’t get a bootable kernel no matter what I did w/ CentOS (and yes, I did load the CentOS .config file into menuconfig before compiling my kernel.)

  Come to find out, because CentOS is using a modified 2.6.18 kernel that little .config file is very out dated. As of kernel 2.6.31 a new kernel parameter is available called “enable deprecated sysfs features to support old userspace tools” which is under the General setup area. Once I enabled that option, I was good to go. It has something to do with why my SAS RAID card wasn’t getting picked up upon boot up. Everything else after that was a piece of cake!

You could have also just added this to your .config file. (NOTE: Hats off to Vanecka!)
CONFIG_SYSFS_DEPRECATED_V2=y

Couple Kernel compile links for CentOS 5:

http://wiki.centos.org/HowTos/Custom_Kernel  (official CentOS wiki kernel build page)
http://www.howtoforge.com/kernel_compilation_centos_p2  (HowtoForge article)
http://kerneltrap.org/node/1783 (Old kernel trap article which mentions the use of -j8 for make parameters so you’ll use all of your cores on compiles! I did a ‘make -j8 all’ and finished my kernel compile in under 5 minutes on a Xenon E5620 CPU!)
http://www.linuxfromscratch.org/lfs/view/6.4/chapter08/kernel.html (LFS link on kernel compile)

Moving lots of data:I do all sorts of networking related stuff and one of the few things I do often are server migrations. So eventually, there comes a time for me to copy a large amount of data from one server to another while maintaining the file permissions and structure. That’s where tar & ssh come into play. If I’m moving a users’ home directory from one server to another I usually use tar & ssh together via a pipe and let it copy off the files perfectly.

tar zcvf - jsmith | ssh "destinations IP" "cd /home/; tar zpxvf -"

Within seconds of the command starting off, you’ll be asked for the password for your ssh session. Then, the data will get flung across the network to the destination folder. Works great!

Verifying Apache Configuration:When adding additional Virtual Hosts to your Apache config file it’s nice to make sure they’ve been picked up correctly (or! that you’ve modified the right httpd.conf file!) A simple way of letting Apache tell you what it’s going to use is via the -t option.

httpd -t -D DUMP_VHOSTS

It parses the running configuration and dumps the virtual hosts a nice readable format. You can also use “DUMP_MODULES” to see a list of the modules it will use which can be useful for whether or not that database module is going to be loaded or not.

MySQL database dump:It never fails, when doing a migration you’re going to need to dump your databases from a MySQL DB into another DB on your newer system. If they’re the same version however you can run mysqldump w/ a couple command-line switches and then import them on the new server.

mysqldump -u root -p --databases "your DB here" > dbfilename.sql

After you’ve moved your database file to the new server you can then import it in via:

mysql -u root -p -D "your DB here" < /path/to/dbfilename.sql

Keep in mind, we’re assuming you’ve already created the database inside your new MySQL server and will assign it proper permissions so your end user can connect to it!

Just a couple quickies for today’s post.

As always, a couple useful links:

http://www.debuntu.org/how-to-create-a-mysql-database-and-set-privileges-to-a-user

http://www.webmasterworld.com/forum49/723.htm

http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html

http://dev.mysql.com/doc/refman/5.0/en/default-privileges.html

This is the second part of my FTPES, vsftpd & CentOS article. In the first part I walked you thru how I built a certificate, sent the CSR off to my CA and finally, modified the vsftpd.conf file. In this part I’ll show you how to test the service via command-line so you can actually see the certificate and how to punch a hole thru your FW because the data part of your FTP session is now encrypted.

First, let’s talk about verifying the configuration. One of the obvious things you can do is open up a command prompt session and attempt to log into your ftp server. Once connected, does it accept an anonymous connection? Was that what you wanted? Also, was the plain jane ftp command allowing you to log in or did you get a 331 error, “Non-anonymous sessions must use encryption.” If you’ve configured it correctly it shouldn’t allow you to login at all. So how do you test it using encryption? By using openssl w/ s_client of course! Use…

openssl s_client -starttls ftp -connect yourserver.example.com:21

This will allow you to not only log into your server using encryption via command-line but also verify that you’ve got the proper certificate & certificate chain installed. I personally tried connecting w/ out the starttls option but wasn’t successful (instead I get a “SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol” error message. I beleive this is because I’m using vsftpd in explicit mode not impicit mode.) The openssl s_client command can also be used for a number of other encrypted services for debugging certificates which is extremely helpful!

So you’ve verified that FTPES does work on the local LAN but doesn’t seem to work thru your FW. This is where passive ftp comes into play because your FTP traffic is now encrypted and your FW can’t do an inspection to determine which port is going to be used next and open it up for you ahead of time. Most FWs come with some sort of FTP inspect feature but we just killed it because that data is totally encrypted now and it’s unable to sniff the FTP traffic and sense which port needs to be opened. You won’t have to turn on passive FTP because it’s on by default with vsftpd. You should specify the port range and I also turned on pasv_address too:

pasv_enable=YES
pasv_addr_resolve=YES
pasv_address=yourserverhere.example.com
pasv_min_port=63000
pasv_max_port=65535

Once done, this will force your vsftpd server to use a port range of 63000 to 65535 for data connections. Alot of the commands I used above can be found w/in the man page for vsftpd.conf (along with their default values!)

Now you just need to open up that same range on your FW device (NOTE: Each FW is going to be different!) On my Cisco Adaptive Security appliance (ASA) I’m defining an object-group first for the port range like such:

object-group service ftp_passive tcp
  description ftp passive range
  port-object range 63000 65535

Then I use that port range in my outside ACL for inbound traffic like such:

access-list outside_access_in extended permit tcp any host YOUR_PUBLIC_IP object-group ftp_passive
.
.
.

Just be sure to use your public IP address for ‘YOUR_PUBLIC_IP’ in the above ACE. Now when it comes to ACL rules there’s tons of different ways to allow traffic so I won’t go into much more detail here other than how I punched a hole thru my FW to allow the encrypted vsftpd data channel traffic through.

That’s about it folks. Remember, if you’re having problems with your configuration, break it down into simple pieces and troubleshoot it that way versus trying to eat the entire problem at once!

-Q

Oh, almost forgot. Here’s a couple links that might be helpful!

http://en.wikipedia.org/wiki/Ftp
http://en.wikipedia.org/wiki/FTPES
http://it.toolbox.com/blogs/unix-sysadmin/troubleshooting-ssltls-mail-services-29266
http://www.wowtutorial.org/tutorial/26.html
http://vsftpd.beasts.org/
http://www.openssl.org/docs/apps/s_client.html
http://blogs.iis.net/robert_mcmurray/archive/2008/11/10/ftp-clients-part-2-explicit-ftps-versus-implicit-ftps.aspx

So this past weekend I went to visit my Aunt & Uncle in Harrisonburg, VA. While there, my Aunt tells me she can’t send email thru my server which I was fairly certain was once again, “user error.” Come to find out. It was a misconfiguration on my part! Ah ha!

Long story short, I had reconfigured my email server about two months ago and added a DNSBL (zen.spamhaus.org) to my sendmail features and it was blocking her from relaying email thru my sendmail server. I simply had to add…

FEATURE(`delay_checks')

… to my sendmail.mc file and restart the service. (NOTE: On CentOS 5.0 if you restart the sendmail service it will check your sendmail.mc & sendmail.cf files and do a ‘make -C /etc/mail’ for you if the sendmail.mc is newer.)

Now, Sendmail will wait on the DNSBL and give the user a chance to authenticate first which will allow them to then bypass the DNSBL. Seems her residential comcast IP Address was added to zen.spamhaus.org which is commonly done to blocks of residential user IP address spaces to prevent spam.

Ask anyone that knows me and they’ll tell you I love Linux. In particular, I love Red Hat’s flavor of Linux. I’ve still got my first Linux book ”The Complete Red Hat 5.2 Installation Guide” and while I don’t consult it first for all problems I do remember thumbing thru its pages when trying to configure those tulip.o drivers and how to change my X Windows config file settings (NOTE: I still to this day get tripped up on configuring X Windows!) So that having been said, it’s no wonder I’m using CentOS and have been for years since Red Hat has stopped releasing a free desktop variety for users to use (I know they’ve got Fedora but that product lines’ release cycle is too aggressive for my tastes.) 

  So that having been said, I love CentOS. I’ve been using it at home for a while now and whenever possible do my best to stick w/ the stock rpms so updates go smoothly. I’ve recently been trying to get vsftpd daemon to use encryption while still jailing my end users in their home directories and here’s what I found out. Vsftpd doesn’t natively do sftp but it does do ftpes pretty well. So I’m going to walk thru generating a private key, making a certificate signing request, adding the public key from your third party certificate authority (CA) to their certificate bundle to form a chained public key and finally, configuring vsftpd to do ftpes.

First, I logged into the box and grabbed superuser.

su -

Then changed directory to the location of the certificates.

cd /etc/pki/tls/certs/

Next, I ran openssl to generate my private key w/o a passphrase! (NOTE: If you decide to use a passphrase a number of services won’t auto start on a power failure which means you’ll need to run home, start the service manually again and enter in the passphrase so the service can use the certificate. Some services can be configured to manually put the passphrase in for you but that’s lots of effort.)

openssl genrsa -out myserver.mydomain.private.key 2048

Had I wanted to generate a private key w/ a passphrase it would have been…

openssl genrsa -des3 -out myserver.mydomain.private.key 2048

You’ll notice at the end of the command I put 2048. That stands for how many bits this certificate is generated with. A number of CAs will require now a days that you use a 2048 bit key versus the old standby of 1024 bit ( the CA I use GoDaddy is one of them.)

The very next thing we need to do is lock down your private key. This is used to encode encrypted communications to the end user who will now use your public key to decrypt the data. So let’s make it readable only by root.

chmod 600 myserver.mydomain.private.key

Once that’s done, it’s time to generate our certificate signing request (CSR) that needs to be passed to your CA so they can generate a public certificate for you.

openssl req -new -key myserver.mydomain.private.key -out myserver.csr

At this point you’ll be prompted for answers to a series of questions which will be placed in your CSR. The big thing you need to get right is your Common Name (CN) question. Make sure the CN is the server name that matches up w/ your DNS entry for this server. Folks are going to use DNS to access your server so the certificate needs to match up with what they are going to use to hit your server across the internet. It’s interesting to note, that w/ GoDaddy, pretty much all of the other fields you fill in get thrown out in the generated public certificate they send back to you so don’t get too freaked out with putting your location information in there. Again, this is with GoDaddy. Other CAs may publish that information w/ the public certificate.

After you’ve submitted your CSR to your CA. You’ll have to go thru the whole domain verification process which can be its’ own little tap dance. I know when our company had their domain names register w/ someone else besides GoDaddy it took 24 to 48 hours for whois verification and whatnot. Once we moved our domain names to GoDaddy the verification process took less that 5 minutes. Coincidence? Business strategy? I don’t know but I’m sure glad we moved our domains over to GoDaddy.

Okay, so you’ve finally gotten your certificate from your CA. Yay! Now comes the fun of lumping that certificate w/ the included CA bundle they sent you and configuring your vsftpd daemon.

First, concatenating the two files together into a combined public certificate file. Why? Cause vsftpd doesn’t have an option for the certificate chain file your CA gave you w/ your public certificate.

cat myserver.crt CA_bundle.crt > myserver.mydomain.public.key

This will put both files together into a combined file which will have all certificates. Now we can configure vsftpd and be done.

Open up the vsftpd.conf file w/ vi.

vi /etc/vsftpd/vsftpd.conf

When looking at your vsftpd.conf file I started at the beginning and slowly worked my way thru turning off features I knew I didn’t need. One of the first was anonymous_enabled. It defaults to enabled unless you uncomment the line and tell it NO. Most everything else I left as is. It’s interesting to note, that the SSL features you’re looking for in the config file aren’t present so you’ll need to add them manually to the file. Here’s what I’ve put in mine. Most of these commands by the way are covered in your man page for vsftpd.conf so take a look there for a more detailed explanation.

ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
rsa_cert_file=/etc/pki/tls/certs/myserver.mydomain.public.key
rsa_private_key_file=/etc/pki/tls/certs/myserver.mydomain.private.key

This essentially turns on SSL, turns off anonymous SSL, forces both data & logins to go over SSL and only allows TLS encryption. It also points to the location of your public & private keys.

This concluded part 1. Part 2 will talk about the implications of encrypting your FTP traffic and how it will mess w/ your FW rules at your border device.

-Q

Links I found useful while doing my install:
http://wiki.vpslink.com/Configuring_vsftpd_for_secure_connections_%28TLS/SSL/SFTP%29

http://en.wikipedia.org/wiki/FTPS

http://vsftpd.beasts.org/vsftpd_conf.html

This post is out of sequence but I told myself I’d throw it up while it’s still fresh. About a month ago I redid my entire email server from scratch when I started having some hardware issues. This post is on one of those bug-a-boos of switching from a physical server to a virtual world. TIME. Seems keeping vmguests synchronized with the correct time is an issue and wrecks havoc upon Dovecot (even when vmware tools are installed!) So, one of solutions was to not only install the vmware tools on my vmguest but to also install ntpd. This is good and bad…

  See, ntpd checks the time rather frequently and all it takes is a sync of > 75 seconds and Dovecot kills itself. Thankfully, it left a message in the log file.

dovecot: Time just moved backwards by 109 seconds. This might cause a lot of problems, so I’ll just kill myself now. http://wiki.dovecot.org/TimeMovedBackwards: 1 Time(s)

The URL provided in the error message was very illuminating and gave some recommendations from there. I installed the script that used lsof and was pretty much good to go. I did a couple of tests to ensure it wasn’t just restarting it once per minute then did the old crontab -e command and placed the script in the root’s cron jobs file stored at /var/spool/cron/root. Life is good. I’ve always been a big fan of netstat but lsof also gives some very valuable information on tracking down rogue issues.