My plans for a Debian file server

While I do have systems filling multiple roles I intend to redesign most of the technological infrastructure inside of my home.  The first phase is to setup the required servers to handle the tasks I have set out for phase one.

The first machine to go online will be a machine called kirk, it will act as the central nervous system for the whole setup, it will need to offer light redundancy at the fastest possible speeds within my budget.  The primary focus of this machine will be NAS, secondary functions are important as well such as an outside facing web server, internal DNS resolution and other interesting services.

All of the hardware I don't have but need to acquire, I've decided to exclusively use the 4 week to ship vendors from Hong Kong, and China to keep the costs as low as possible.  I find the quality to be acceptable and it always seems to fit my needs.  I'm of course using eBay to fill the voids of hard to find hardware.

The first step in this endeavour will be to assemble and test the hardware.  Currently here is what's planned for kirk (yes, I plan to use Star Trek captain's names as servers):


Hardware list:

MB - Tyan Thunder K8W S2885

Although heavily obsolete since it was originally released in 2003, this board allows a lot of peripherals to be used on it.  For the tasks it is intended to do, it will serve nicely.  The only drawback that I find with this unit is the power hungry attitude that servers took a few years back, though I plan on putting in less power hungry CPUs in it to reduce its idle requirements by almost 50 watts.  This server board can provide up to a GiB of bus width on the PCI-X bus although the PCI-e bus has surpassed those figures, my stance that anything better than the bus maximum of 133 Mb/s on the original PCI bus I'm ok with.

Memory - 8x 333MHz 1GB DDR DIMMs

A total of 8 gigabytes of RAM will be sufficient for caching writes and buffering file transfers.  I plan on using tmpfs to as a ramdisk for some functions, though I will have to determine what those functions will be exactly.

CPU - 2x - S940 Opteron 280

The original CPUs that resided on the motherboard when I obtained it were Opteron 242 chips.  I've decided to order replacements for these chips as they are rated at 85W TDP and consume massive amounts of electricity.  I found the 280's on eBay for $20.00 shipped.  I am still awaiting their arrival so that I can install them into the motherboard with the new HSF's I ordered from ncix.com.

Though not confirmed, the eBay description said the steppings on these chips were of the 68W TDP variety.  I hope that's true as moving to these dual core chips would lower the power draw and heat, as well as speed up the system.  Jumping from 1.6 GHz to 2.4 GHz should certainly be a large enough step up to notice.

HSF - OCZ Vendetta

I have decided to upgrade the coolers on both chips, not because of noise, but because this machine will reside in a non-friendly temperatures environment and it'll need all the cooling power it can get.  I found that ncix and its other websites were offering refurbished models, quickly price matched a couple of them and ended up at around $8.00 a piece.  Good deal.  When they arrived, I inspected the units and found they were in close to perfect condition.  Nothing out of the ordinary to report.

Cooler Master Stacker - Model STC-T01

An old case that I used to have my gaming rig setup in.  It is currently converted to the inverted BTX layout where the motherboard sits on the left of the case.  I do believe this will be well suited for a file server, judging solely by the amount of drive bays that it has avialable.  The cooling potential is excellent as I also have the crossflow fan installed into the case, hopefully this keeps the VRM's on the motherboard at an acceptable temperature.  The best part is that it potentially has 12 available 5.25 half-height bays.  I still have to figure out how to move the buttons for power and reset outside of the top bay.  Options are to drill the top plastic bezel and add momentary switches, or wire the whole shebang outside of the case into another box like a project box or some sort of repurposed box.  I'm still mulling this one over.

2x RAID Controller - 3ware 9500s 4LP

I purchased two of these to expand the total number of hard drives available in the system.  Though I haven't nailed down the exact RAID configuration I plan on using I figured I'd benefit from the expanded bandwidth that PCI-X would provide.  The extra cache is a bonus as well.  I'm torn about using the integrated RAID features mainly due to the bad reviews I've read out there.  The real speedup won't be noticeable as the card is limited to a 66MHz bus at 64 bits wide.  My eyes glazed over after a while and feel that a Supermicro AOC-SAT2-MV8 would have been a better investment, mainly because it connects 8 drives and has a bus speed of 133MHz.  True it's not a hardware RAID card, but I get the feeling that I'll be using MD arrays in Linux anyway.  I don't know, maybe I'll throw them back on eBay after a few file tests.

3Ware Escalade 8506-12

This is the third raid controller I ordered, after realizing that the on-board SATA controller on the motherboard was linked using the PCI bus, I panicked because the whole PCI bus on a motherboard is limited to a maximum throughput of 133MB/s.  Not enough for my needs, hence my "all-out" attitude when I went searching for another expansion card that I could connect the hard drives to.  This is an even older generation card, quite similar to the 9500 series, but no massive cache on-board.  Difficulty with using this is that it possibly negates any advantages I'd get from using the other controller cards.  If my RAID array spanned into this card, the lower write cache buffer, along with a possible 12 other hard drives fighting for attention may starve the drives.  Something I was wholeheartedly attempting to avoid.  This card is of the same status, not really sure if I'll end up using it.

So, the total available SATA ports are 24, of which I plan on using 16 from the get-go, the option to expand externally is an path I may take down the road using infiniband and an external case to create an external storage array.

4x - CM 4in3 Hard Disk module

Although I have a few of these kicking around, I'm still contemplating if I should be using all 4, as the real limit is having a way to move the power and reset switches out of the Cooler Master case so that I can make use of all 12 bays in the case.

Hard drive(s)

Hard drives, what can I say about them except that I have a multitude of different size and manufacturers of these beasts.  From 160GB all the way up to 2TB.  Torn currently on the actual configuration I plan on running with due to the uncertainty of the controllers I plan on using.  I know one thing for sure, all must be 7200RPM or better, not necessarily due to the idea of faster reads/writes, but the reliability of those so-called green drives seems to be in question, especially in a RAID setup.

Well, this is what I have so far, I am in the process of putting together the hardware and awaiting some parts to consider moving this project along.  The one niggly thing I plan on doing is setting up a mirrored pair of Compact Flash cards as the boot device.  Due to the write cycle issue that exists with flash devices, I am unsure how to proceed with the setup.  While I can make the system run, and use these as the main source of binary goodness, there are a couple of directories that Linux constantly updates, all well and good, simply use tmpfs, good no?  Well kind of, there are two directories that I am unsure what to do with, one is /var/log, the other /var/tmp.  The first, I simply want to keep my logfiles intact, I certainly like to know what's going on, and they're really useful if you want to play the blame game when things go awry.  The other, well, the FHS states that this directory is intended for the purpose of storing temporary files that will survive a reebot, or system downtime.  ie, those files are intended to be there at bootup.

My initial approach was to either a). assign a single dedicated drive for this task or, remoely log everything via rsyslog.  b). write those directories to the CF cards (bad idea, don't even know if they employ wear-leveling.  If I were to save the logs residing in memory, then I could pull an event or trigger to flush those writes to disk every so often.  I think this may be the best solutions I can  come up with at the moment.  Feel free to chime in if you have any suggestions.

Thanks for reading!