Posts Tagged ‘SCST’

CentOS 6 with SCST

I keep meaning to write down my SCST installs so the other network engineers here at work can reference how I did it all. This documentation is also for me too (well, primarily for me actually!)

Here it goes…

1.       install CentOS:

a.       While on Hostname screen, select “configure Network” and check “connect automatically” on your eth0 device so it will install & configure basic networking alongside NetworkManager.

b.      Select the “Software Development Workstation” option then unselect groups virtualization, virtualization-client and virtualization-platform or let the installer finish then use “yum groupremove virtualization virtualization-client virtualization-platform” once you’ve booted into the OS the first time. By selecting the “Software Development Workstation” as your install type, it will pretty much install every package you’ll need to do the kernel compile.

2.       Now do a yum update and reboot the box so you’re using the new kernel.

3.       Install some remaining dependencies for this full kernel build:

a.       yum install asciidoc

b.      yum install newt-devel

4.       Commence kernel build prep.

a.       Find out which kernel you’re currently using w/ “uname -r” and write it down.

b.      Create user for building kernel & SCST by running “useradd mockbuild” go ahead and change from user root to that user “su – mockbuild” for the next step (no, mockbuild doesn’t need a password so don’t give it one!)

c.       While as the mockbuild user, execute “mkdir -p ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}” which creates the directory structures for the build and finally “echo ‘%_topdir %(echo $HOME)/rpmbuild’ > ~/.rpmmacros” which creates your .rpmmacros file w/ the build directory location. (NOTE: spacing is very important on this command. Just copy & paste to be sure you typed it correctly.)

d.      Again, while as the mockbuild user, execute “rpm -i http://vault.centos.org/6.4/updates/Source/SPackages/kernel-2.6.32-358.14.1.el6.src.rpm 2>&1 | grep -v mock” which will install the current kernel source and put the files in your build tree folders (while also redirecting stderr to stdout while grepping it.)

e.      While in your homedir and as the mockbuild user, execute “rpmbuild -bp –target=$(uname -m) ./rpmbuild/SPECS/kernel.spec” so it will uncompress the kernel sources and move them to your ./rpmbuild/BUILD folder. (NOTE: While it’s running, open up another console and as root run “rngd -r /dev/urandom” as the certificates will drain out the random number generator and kind of hang the rpmbuild.)

5.       Download SCST from svn & patch kernel in BUILD directory.

a.       We need to grab a copy of SCST using svn. There’s a number of branches but we want the latest stable. So run “svn co https://svn.code.sf.net/p/scst/svn/branches/2.2.x scst” while the mockbuild user and in /home/mockbuild directory. This will
be the latest stable version of SCST which is 2.2.1.

b.      Now that our user mockbuild has a copy of scst in his home directory, go ahead and cd into the unpacked kernel so we can patch it. Should be “cd ~/rpmbuild/BUILD/kernel-2.6.32-358.14.1.el6/linux-2.6.32-358.14.1.el6.x86_64/”.

c.       We need to patch for SCST’s fifo vs lifo: “patch -p1 < ~/scst/scst/kernel/rhel/scst_exec_req_fifo-2.6.32.patch”

d.      We need to patch for iscsi-scst’s put_page_callback: “patch -p1 < ~/scst/iscsi-scst/kernel/patches/rhel/put_page_callback-2.6.32-220.patch”

6.       Time to compile the Linux kernel & install it.

a.       Check your .config files (defualt one inside your ~rpmbuild/BUILD/kernel-*/linux-*/.config to your current kernel config in /boot/config-2.6.32-*) They shoud pretty much match. This means you can skip “make oldconfig” which would prompt you for stuff missing from your provided .config file with what this kernel you’re about to compile has as of now (meaning, if you’re compiling a newer kernel, your .config file is missing some stuff so it will prompt you for defaults and update your older .config file!)

b.      Change the kernel name by editing the Makefile. We need to separate this kernel from the others by editing the EXTRAVERSION varible. Change it to “=-scst” version versus being just empty or blank.

c.       Now run “make menuconfig” and change the following settings for SCST:


i.  Select “Network support”->”Networking options” make sure “TCP/IP networking” is checked (should be) and check “TCP/IP zero-copy transfer completion notification” underneath so SCST will be notified when a transfer of data across the nic has finished. This way, it can hurry up & send the next block. Should speed things up.


ii.   Next, verify that “Device Drivers”->”SCSI device support”->”SCSI disk support” is already checked. Again, for RHEL 6.4 it should be.


iii.  Verify that “Enable the block layer”->”IO Schedulers”->”CFQ I/O scheduler” and “Default I/O scheduler” set to CFQ are both set. Again, they should be.


iv.  Lastly, set “Processor type an features”->”Preemption Model” is set to “No Forced Preemption” versus Desktop mode.


v.  Once done, save your config so we can compile this kernel.d.      Once that’s done, we can compile it. Type “time make -j8 all” as the mockbuild user. This makes the kernel & modules while timing it and using all of your CPU’s cores (for me, I’ve got 8 cores so -j8!)

e.      Now we need to install our new kernel. Do a “make modules_install” as root which will copy the modules over to /lib/modules/<kernel name> directory (you did call your kernel something different by setting the EXTRAVERSION variable right?) After it’s done, do a “make install” as root as well which will update the /boot directory & grub. Now update grub.conf so it will default to our new kernel upon boot up.

f.       Once that’s done, copy over the current .config file in the /home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-358.14.1.el6/linux-2.6.32-358.14.1.el6.x86_64 directory to the /boot directory and name it such that it matches your other config files in that directory (such as config-2.6.32-scst.)

7.       Reboot your system so we can use the new kernel and compile scst off the new kernel source files.

8.       Now we’ll compile SCST.

a.   Go ahead and change the various Makefiles at once to release version by issuing “make debug2release” in the main scst directory. This will remove the debugging stuff. Do this as root. Also, since CentOS 6 now uses sysfs and not proc, we don’t have to do a “make enable_proc” like for CentOS 5.

b.      Now go ahead and issue “make scst scst_install” as root in the mockbuild directory’s scst folder.

c.       Issue “make iscsi iscsi_install” as root in your current folder.

d.      Finally, issue “make scstadm scstadm_install” as well. This finalizes the compiling part.

9.       Configure & start SCST.

a.       Need to go ahead and tell our system to start it up automagically on bootup so execute “/usr/lib/lsb/install_initd scst” as root user.

b.      Go ahead and either reboot to test it or “service scst start” to fire it up. (NOTE: We haven’t done a config file yet so it’s running w/ nothing up!)

c.       Next, we need a simple config. Since I’m using actual RAID volumes as disks in SCST, I’ve gone w/ BLOCKIO devices and list them as /dev/sdX devices. You may want to use FILEIO devices instead or even a different device handler all together (scst_disk instead of scst_vdisk.) Here’s a couple of different examples:


i.      dev_disk example which uses scst_disk:

                                                HANDLER dev_disk {

                                                                                DEVICE 3:0:0:0

                                                }

                                                TARGET_DRIVER iscsi {

                                                                                enabled 1

                                                                                TARGET iqn.2013-07.com.example.san1:iscsi-tgt1 {

                                                                                                                enabled 1

                                                                                                                LUN 0 3:0:0:0

                                                                                }

                                                }


ii.     vdisk_blockio example which uses scst_vdisk:

                                                HANDLER vdisk_blockio {

                                                                                DEVICE disk01 {

                                                                                                                filename /dev/sdb1

                                                                                }

                                                                                DEVICE disk02 {

                                                                                                                filename /dev/sdc1

                                                                                }

                                                }

                                                TARGET_DRIVER iscsi {

                                                                                enabled 1

                                                                                TARGET iqn.2012-09.com.example.san1:iscsi-tgt1 {

                                                                                                                enabled 1

                                                                                                                LUN 0 disk01

                                                                                                                LUN 1 disk02

                                                                                }

                                                }


iii.     vdisk_fileio example which also uses scst_vdisk:

HANDLER vdisk_fileio {

                                                DEVICE mydisk01 {

                                                                filename /home/testdisk1

                                                                nv_cache 1

                                                }

}

TARGET_DRIVER iscsi {

                                enabled 1

                                TARGET iqn.2013-07.net.spiderbird.san2:iscsi-tgt1 {

                                                                enabled 1

                                                                LUN 0 mydisk01

                                }

}

10.   Finalizing your install:

a.       Now that we’re done, it’s best to configure yum not to update your kernel w/ a newer one which would break all of your hard work. Modify your /etc/yum.conf file and add “exclude=kernel*” to it w/o the double quotes. This will help keep your system somewhat up-to-date w/o breaking SCST & your custom kernel.

11.   Troubleshooting:

a.       Keep in mind, if you’re going to use multiple SCST SANs in your backend, you need to make sure all of your exposed LUNs have different t10_dev_id values else your host will get confused and assume it’s seeing different paths to similar items. Also, using setup_id is supposed to help prevent this but evidently doesn’t work in version 2.2.x. You can skate around this issue by using different names for your defined DEVICEs (instead of using disk01 try using SAN1disk01.)

b.      Pass through devices work as expected except for when that pass through device is a simple SATA drive. Your ESXi 5+ server still won’t put a VMFS partition on there because it isn’t reliable storage. Best to stick w/ RAID volumes in pass through mode instead.

c.       I forget C….

Here’s a couple of links that I used during this write up:

Kernel compile links:

http://wiki.centos.org/HowTos/I_need_the_Kernel_Source

http://wiki.centos.org/HowTos/Custom_Kernel

SCST info regarding speed, settings & setups:

http://scst.sourceforge.net/max_outstanding_r2t.txt

http://scst.sourceforge.net/iscsi-scst-howto.txt

http://scst.sourceforge.net/vl_res.txt

http://sourceforge.net/p/scst/svn/4920/tree/branches/2.2.x/ (SCST SVN)

CentOS FAQ regarding install groups:

http://wiki.centos.org/FAQ/CentOS6#head-ef37a027209dcdba0f8f00cf2696721b774f74bf

A bunch of different write-ups for various Linux installations:

http://www.howtoforge.com/how-to-create-a-fiber-channel-san-using-scst-with-qlogic-hba-on-linux-debian-6  Debian…

http://aliuhui.blogspot.com/2012/04/setup-iscsi-target-with-iscsi-scst-20.html  Ubuntu …

http://jaysoncena.wordpress.com/2013/03/17/26/  generic..

http://temporalsolutions.com/linux-ha/scst-setup/ Ubuntu..

http://wiki.alpinelinux.org/wiki/High_performance_SCST_iSCSI_Target_on_Linux_software_Raid  Alpine Linux

http://monklinux.blogspot.com/2012/02/scst-configuration-how-to-using-gentoo.html Gentoo

http://www.bauer-power.net/2012/03/how-to-install-scst-on-ubuntu.html Ubuntu..

http://www.kcsts.co.uk/blog/scst-iscsi Ubuntu again…

http://www.lessfs.com/wordpress/?p=850

http://www.lessfs.com/wordpress/?page_id=577

http://marcitland.blogspot.com/2011/03/accelerating-vdi-using-scst-and-ssds.html Gentoo

Here’s a number of links regarding iSCSI settings you can tweak and what they mean. These are primarily geared towards using your SCST installation with ESXi and MPIO:

https://communities.vmware.com/docs/DOC-15437

http://deinoscloud.wordpress.com/2010/08/19/vsphere-4-1-iscsi-advanced-settings-and-their-meanings/

http://forum.open-e.com/archive/index.php/t-2537.html (Tuning iSCSI settings)

http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vsphere.html Best link I’ve found yet of tweaking your ESX iSCSI settings.

http://www.gatt-it.co.uk/1/post/2012/11/how-to-set-the-round-robin-iops-setting.html

http://forum.open-e.com/showthread.php?1542-VMWare-vSphere-Recommended-settings

https://forums.openfiler.com/index.php?/topic/4681-howto-definitive-esxi-41-configuration-guide/

 

 

 

 

Custom Vanilla kernel part 2…

So I’ve expanded my SCST compiles onto other Linux OSes namely Fedora 14 & Red Hat 6.0 and I’ve come across another issue when attempting to get the modified kernel to compile, namely it doesn’t pass the precompile check and never fully assembles into a binary! This one took me a while to figure out but thankfully Google provides for those that seek…

The errors I was seeing were similar to below:


linux/X64/usr/include/drm/drm_mode.h:85: found __[us]{8,16,32,64} type without #include
linux/X64/usr/include/drm/i915_drm.h:120: found __[us]{8,16,32,64} type without #include
linux/X64/usr/include/drm/mga_drm.h:260: found __[us]{8,16,32,64} type without #include
linux/X64/usr/include/drm/radeon_drm.h:758: found __[us]{8,16,32,64} type without #include
linux/X64/usr/include/drm/via_drm.h:117: found __[us]{8,16,32,64} type without #include
linux/X64/usr/include/scsi/scsi_netlink.h:108: found __[us]{8,16,32,64} type without #include
linux/X64/usr/include/linux/resource.h:49: found __[us]{8,16,32,64} type without #include

Essentially a precompiler option #include was messing things up. Thankfully someone over on kernel.org found it and knew what to do about it. You’ve got to do your make menuconfig, make your new .config file then edit it w/ vi. Search for a line stating CONFIG_HEADERS_CHECK=y and make it =n instead.

After that it was smooth sailing and the kernel compiled just fine on Red Hat 6.0 & Fedora 14. I eventually had another problem w/ Fedora 14 down the road but installing redhat-lsb via yum solved that problem (had to do w/ the initial functions that are included in Linux Standard Base!)

Here’s the link I found regarding the compile issue:

https://patchwork.kernel.org/patch/274141/

Custom Vanilla kernel for CentOS

I’m in the process of building up a SAN from scratch using SCST & CentOS. I’ve played around w/ kernels in the past but hadn’t done it in a while. When downloading 2.6.36.2 from Kernel.org recently I couldn’t get a bootable kernel no matter what I did w/ CentOS (and yes, I did load the CentOS .config file into menuconfig before compiling my kernel.)

  Come to find out, because CentOS is using a modified 2.6.18 kernel that little .config file is very out dated. As of kernel 2.6.31 a new kernel parameter is available called “enable deprecated sysfs features to support old userspace tools” which is under the General setup area. Once I enabled that option, I was good to go. It has something to do with why my SAS RAID card wasn’t getting picked up upon boot up. Everything else after that was a piece of cake!

You could have also just added this to your .config file. (NOTE: Hats off to Vanecka!)
CONFIG_SYSFS_DEPRECATED_V2=y

Couple Kernel compile links for CentOS 5:

http://wiki.centos.org/HowTos/Custom_Kernel  (official CentOS wiki kernel build page)
http://www.howtoforge.com/kernel_compilation_centos_p2  (HowtoForge article)
http://kerneltrap.org/node/1783 (Old kernel trap article which mentions the use of -j8 for make parameters so you’ll use all of your cores on compiles! I did a ‘make -j8 all’ and finished my kernel compile in under 5 minutes on a Xenon E5620 CPU!)
http://www.linuxfromscratch.org/lfs/view/6.4/chapter08/kernel.html (LFS link on kernel compile)

Return top