[nmglug] Linux built SAN

[Nick Frost]

Andres Paglayan wrote:
> thanks,
> we are upgrading to a gig switch,
> and I am not versed enough to determine about SAN vs NAS,
> I kinda imagined SAN would be easier to set up,

Andres;

Actually a NAS (networked attached storage device) is much easier to 
set-up because you don't have to design a SAN topology (2nd network) and 
hook it up to your existing network.  If the Linux box in question has 
Gigabit Ethernet, the easiest thing to do is assign it an IP and make it 
a NAS device (file server).  The premier NAS devices (IMHO) are made by 
Netapp (www.netapp.com) but they are very expensive. You can get a Linux 
based NAS from Infrant for $1,200.00 that has a storage processor 
optimized for NAS/file serving, but since you have a 3ware 9500-S card, 
you might as well roll your own, though I think you'll want a PCI-X slot 
depending on which card you have, and that will affect the type of 
motherboard you will want to plug it into. At the Santa Fe Institute, we 
used a 3ware 9500-S12 on  a Tyan K8W (S2885).  There is a nice article 
(albeit for Solaris x86) here, which provides some reference for the 
variables involved in cooling 12 drives in a case, etc.

http://blogs.sun.com/PlasticPixel/entry/build_your_own_multi_terabyte

Though the CM stacker cases can be purchased from Newegg, I think the 
drive bay units cannot (any longer), he used the Cooler Master drive bay 
units, which can be found here;

http://www.coolerguys.com/840556033615.html

There are some Linux variants for NAS setups, like NASlite

http://www.serverelements.com/naslite.php

but, for that matter, I think it's probably just as well to set-up the 
server with the 3ware 9500-S and disks, then simply configure the 
daemons that you want depending on the clients.  You'll probably want 
SMB/CIFS for Macintosh and Windows clients, possibly AFP if you have a 
Mac-based network, and NFS for Linux clients.  We have an Infrant 
ReadyNAS 600B ($1,200) (www.infrant.com) and are serving files with 
SMB/CIFS, AFP, NFS, HTTP, and some Rsync (backups).

The main issues (for me anyway) are usually making sure all the SATA 
drives are well-cooled, setting up RAID (choosing hardware or software), 
and establishing Rsync backups (crontab).  If you do hardware RAID, the 
9500-S will allow you a hot spare, and the 3dm/3adm software (Web based) 
for Linux works nicely, and I think I recall it has email notification. 
However, I wrote some BASH scripts for RAID reporting on the SFI mail 
server which had 4 U320 SCSI disks doing RAID 5 on an Adaptec 
controller, and if you want sample scripts for SCSI, I can email those. 
We built a prototype SATA file server with a 3ware 9500-S12 at the Santa 
Fe Institute, and the I/O was most impressive when benchmarked with 
bonnie++ and iometer.  One problem as I see it, however, is that if the 
3ware 9500-S hiccups, you can lose all your data due to proprietary 
encoding; this has not happened to me, but I have heard/read stories on 
the web. From this point of view, some argue that one is better 
configuring the disks in JBOD fashion and doing the RAID in Linux, 
because if something goes very wrong, you can still pull disks, mount 
them in another machine and the data is readable.  mdadm is nice that 
way, in that you can scan a disk and recover RAID signatures/ID's and 
such.  Though I have not tried it, I think you can do double-parity 
software RAID 6 in Linux. The Infrant ReadyNAS 600 that we use here has 
4 disks in RAID 5 configuration (Linux software RAID). Recently we had a 
disk that started to fail, I removed it from the array, turned the NAS 
off, replaced the drive, rebooted, and it rebuilt the array in < 2 hours 
and all was well, no data lost.

I am in the process of preparing a prototype server running Solaris x86 
because I am interested in ZFS, especially its double-parity features, 
and we need a file server for the small San Francisco office. The issue 
of drive cooling has figured prominently in the puzzling I have done 
over this.

Supermicro makes some nice cages for hot-swap SATA, like this one;

SUPERMICRO CSE-M35T-1B Black 5 Bay Hot-Swapable SATA HDD Enclosure - 
Retail
http://www.newegg.com/Product/Product.asp?Item=N82E16817121405

what I don't like about it is the single point of failure on the FAN. 
However, there are not many alternatives.  There are some good coolers 
like the Vantec Vortex 2, but they are $30/apiece and use an entire 
drive bay.

I have though of taking a cheap cooler like this one;

http://www.coolerguys.com/840556058328.html

and replacing the sleeve bearing 40x40mm fans with quality ball bearing 
fans, but that scenario is likely to be A) very loud (server room only 
kinda thing) and B) will consume one 5.25" bay per drive and C) no hot swap.

-Nick
           _      __   ____
    ____  (_)____/ /__/ __/
   / __ \/ / ___/ //_/ /_
  / / / / / /__/ ,< / __/
/_/ /_/_/\___/_/|_/_/
Nicholas S. Frost
nickf at nickorama.com