Resurrecting old hardware...

Hardware is getting faster and faster, so people upgrade as the needs get bigger and bigger. What about the old hardware that stays behind? Some of it gets reused for low priority servers but most of them get thrown away as obsolete. The worst case is if the hardware itself has become obsolete aka its maxed out when it was current.

Well I happened to be involved in one of this type of resurrections. The server was an IBM xSeries 220 tower with 72GB hardware raid 1 array and a small 40GB IDE drive. The idea was to upgrade the small IDE to a 160GB drive to get more storage area. As the 220 has show insane stability the idea was well at the beginning, well until the new drive was installed and well, the Linux recognized it. But here is where the problems started. The performance was insane - the drive would max out at 500KB/s with the read timings test (hdparm -t /dev/hda). The other symptom was drive seek errors which would indicate that there were bad blocks at the beginning or supernodes which would mean that the drive was dieing.

As the Maxtor's drive was brand new (just out of the anti static bag) it could not be the case. After a test with a second 160GB drive the results were the same. Logically thinking - this means we have hit the LBA32 127GB limit. The BIOS cant address more address space than 127GB and when you do you hit the brick wall of buffer overflows in the address registers (most likely when you write over 127GB you start from the 0x0 h again and write up the drives beginning). Of course the Linux kernel takes over the drive addressing after you boot up, but for that to work well the BIOS has to be satisfied first. Well copying 127GB at the rate of 500KB/s would take 266 338 seconds - thats around 3 days straight - I didn't have that much time to wait to see what would happen after the limit gets exceeded.

So what to do when your hardware is too obsolete for upgrading to your needs? You hack it!

In my case it was simple as Maxtor has firmware utility called SeaTools that can change the drive capacity in the firmware. Why is that good you ask? Well this way we can fake the hardwares without using the soldering iron or assembler to hack the firmware in the device.

The procedure goes like this:

1. install the new HDD
   
2. boot into BIOS and accept the changes (new HDD 6GB?!?!?)
3. boot into the SeaTools utility (CD or floppy)
4. Change the drives capacity to be 32GB
5. Shutdown the machine and remove the drive
6. Boot into the BIOS and accept the changes (the IDE drive has gone away)
7. Shutdown the machine and reconnect the drive
8. Boot into BIOS and accept the changes (new 32GB drive)
9. Boot into the SeaTools and restore the drives capacity
10. Boot into Linux :)

The BIOS still thinks it has a 6GB drive connected but its happy with it. Also the Linux boots up faster as BIOS is not complaining about seek errors when addressing the drive.

Running hdparm  -t should result in 13...25MB/sec - not the best speed but a lot better than 500KB/sec. You may ask what the range is so big - but I don't have an answer. On some systems the drive runs with udma2 mode on others in mdma2 mode. Forcing the mdma2 mode to go to udma2 mode did not seem to have the desired performance change but still its working well enough.

Many may ask "why would you ever want to resurrect an old xSeries 220", but the answer is simple. The system has been in use for a long time and if it has not died so far it wont in the near future. Also the speed is boosted with built in SCSI drives.

So where to use the old xSeries 220? Well, for test servers, for low priority web/sql servers, dns servers - there is no good answer. It all depends on the needs of the system. One place would be central log server - aka other servers log to this host, the syslog would save the logs in real time to the SCSI drives and nightly you rotate the logs to go into the mysql database on the IDE drive. Why do this? Well, logs are here for one purpose to be analyzed afterwards.

Another good example would be to use it as a file server - share out the data on SCI to Linux hosts and from IDE to windows hosts and if they start to complain that the speeds are soooo slooowwww blame it on windows and propagate the move to Linux :P

What can I say - use Your imagination.
Tested on CentOS 4.5 - but should work on any Linux with 2.6 kernel.

Blogged with Flock

LVM volume management made easy...

As a enterprise architect/infrastructure developer I'm faced with allot of problems that seem trivial on small scale but when you need to manage thousands of workstations or servers become critical problems. One of them is hard drive partitioning.

For security and stability reasons it is wise to split your root file system to several peaces - my default is:

/boot
/
/tmp
/var
/var/log
/home
swap

This may seem like crazy layout, but when you think of it, most problems start with servers getting root kited by some trivial exploit that will be used to trigger a buffer overflow and there for get a shell. If the tmp, var, var/log and home are mounted as noexec even when the exploit manages to download the overflow script - it wont be allowed to execute. This means its 1 step closer to avoid a security breach.

As I said before - it looks like a trivial task - install the server properly and thats it. Well this advice is good for up to 25..50 machines per admin, but it will fail miserably when you are running hundreds or thousands of machines. You may succeed to install them but soon after you are faced with a problem that some partitions need to be resized because one is almost empty and the other is constantly filling up.

If its just a pc you can take it offline and reinstall/boot to rescue and resize them. But when you are running mission critical servers its not an option. The same gos when you are running systems with different hardware configurations and there fore with different disk sizes. Here comes in the Logical Volume Manager (LVM). It creates another layer of abstraction on top of the physical layer therefore giving you another layer of dynamic configuration.

The greatest advantage of using LVM is that it you can create/resize/delete volumes on the fly without needing to reboot the server or boot to rescue to manage it. Sure it is not as simple as it sounds but basically it works wonders. It also gives the possibility to build the LVM volume on top of the hard drive or a raid device on full extent leaving you (in ideal case) with only 1 partition on the device (I use 2 - I like to keep /boot separately).

So in my case it is not wise make "full" partitioning of the drive, but to keep the partition sizes to minimum (that means with 10% overhead) leaving the rest of the drive free. This will allow me the simply "add" space to the volume that needs space without needing to shrink another volume (this volume cant be mounted while shrinking so this will need rescue mode!!!). Also keeping operating system on one volume group and data on another can save allot of work when restoring a system. For this I use a software designed by me called FabricManager - looks like I have to blog about in the future...

This works well until you run into a problem - You have 1000 workstations installed with this configuration and you want to do a dist upgrade. All is goo until you run out of space in the root volume to do it. 80% of the disk is filled with downloaded rpm's and there is no space left to install them. With LVM it is no problem, You just add some space to the root volume and all works out well, but here comes the catch - how long will it take to manually resize the root volume in 1000 workstations? I'm guessing around a month or two. And this is the best case scenario. Assuming that a good security policy is in place and none of the workstations have the same root password the time could be extended to few years which defies the whole point of upgrading. Sure if all the workstations have been installed with the same root password you can use an script to run the command automatically, but then You will run into about 10% of computers that wont accept the password and the 10..20% of PCs that are turned off at the time you run it. What about strict security policy that wont allow root to even log in remotely? What if the workstations are in different locations or countries or even continents? You can't just run the script few times and hope for the best as some workstations get several times the space needed and some get none.

And so a trivial problem grew out of proportions in seconds and one small design flaw can cost ALLOT of time trying to fix all the problems separately. this led me to write a small shell script that would be able to create/extend logical volumes as needed and it can be found here. It is not 100% what I need but the core functionality is there and it works.

This also solves problems with few "special" applications that need their own partition to run and as I try towards simplicity I can now package the script into the autoupdate package (right... haven't mentioned it before also... need to fix it in near future...) and set a dependence for it. So to automate the install of the application I just add one row in the %pre section of the package and woala - new volume created in the install time :D

It also helps to resize partitions by simply updating the base package of my configuration that contains the sizes of the required volumes. this gives me the ability to reconfigure workstations in few days to new configuration and life can go on...

For conclusion - security is important, so no cutbacks should be made there, noexec and nodev are your friends - use them wisely, LVM is a great tool but it is not a magic bullet - adding an extra abstraction layer multiples the complexity of the system so use it carefully and when developing large scale infrastructure multiply the the count of computers with 1000 and think if the solution is easily managed then.

This script was built for CentOS 4.4 and tested on it - it will work on other distros but may need some modifications.

technorati tags:CentOS, LVM, security, Infrastructure, bash, script, rpm, management, partitioning

Blogged with Flock

x86 on a POWER...

It seems like IBM has few tricks in their sleeves. Normally there would not be an efficient way to run Linux on their mainframes thanks to their fully hot-swap architecture. Fried a CPU or a RAM? Maybe even a mainboard? No problem - they are hot swappable :)

More info on the architecture here. Normal small to midrange enterprises usually don't need them as they can deploy grid architecture, but there is wide range of services that need to maintain around 99.9+% of uptime with high loads. Here comes the System p to fix most of the problems. As grids are powerful, they have some shortcomings. They don't provide enough power for high end calculations as they usually are linked with up to 1gbit interfaces - the system P is on the other hand interlinked with maximum theoretical bandwidth of 30gbit or more interlinks. It also provides the hot swap for all components (well almost all) - I haven't looked into System p design for a long time, but the only part that was not hot swap was the interconnect plate around 3 years ago. Usually these system sun without problems (after they have been set up properly) 6+ years and when the crash happens its total as people who set the system up most likely have moved up the food chain and got a better job or just wont remember how to set them up. Here is where the grid is more fault tolerant - you can loose some of the nodes without any impact on overall performance (and I don't mean "2 pc clusters"). Grid hardware is also cheaper (usually normal PC's or racked server or blades) interconnected with ether 1gbit full duplex single mode fiber, 10gbit multi mode dark fiber, or when the requirements are high enough a profession interconnect.

Anyone wanna donate an IBM System p5 595 with 64x2.3GHZ CPU, 2TB RAM and a 28.1TB storage node for hmm... testing purposes? :)

technorati tags:x86, POWER, System, P, IBM, Virtual, Linux, environment, beta

Blogged with Flock

CentOS 5.0 is out...

The long waited release has happened - CentOS 5.0 is officially out!

The good news is that the base packages are recompiled and are out - extras/addons repos are still empty. the kernel is fairly new 2.6.18 this means the OS has SAS support - yehaa :) Since most IBM 366 are running low on supply, the new IBM 3550 are in (well, it still needs testing a bit, but it at least boots up properly) :) This is a huge step forward but there is still a long way to go. This might give some idea on how long way...

Out of curiosity I tried to upgrade one of my 4.4 servers to 5.0, but it did not go well. After removing around 40 packages I finally got yum to upgrade the kernel and the rest of the system. Well as a result I found out that the upgrade process died in the middle of the process. Rpm had removed its libs and tried to install new ones - it crashed with unresolved deps. Last test was to boot the machine, and it didn't go well also - it crashed on a attempt to load LSI Fusion MPT driver :( As it was a test the results are not promising. Also the lack of apt and apt support for repos (I really really hate yum and up2date). Yum is too talkative and generates several race conditions wen doing the upgrade, so it lacks the needed silence to do scripting!

The conclusion - It works, looks better than 4.4, is more resource hungry (at least 512MB to do a graphical install), lack of support for apt and empty extras-, conttrib-, plus repos, boots faster (didn't take time with stop watch, but it looked faster at least). This means its to crude for enterprise wide adoption, but it shows great promise, next tests will probably be done in few months to see if it has improved.

PS! if any CentOS developers see this post, PLEAS PLEAS PLEAS port apt to CentOS 5, yum just does not cut it and alternatives are always good to have.

technorati tags:CentOS, 5, SAS, IBM, support, released, developers, apt, yum

Blogged with Flock