Central logserver - vol 1 - the basics...

In a normal home environment no one uses a log server as it does not have any practical usage - or does it?

On a enterprise environment it is natural that administrator(s) once in a while look through the servers once in a while (or at least when something crashes they give a quick checkup). In big enterprises it is natural that there is a central log host that collects all the logs into one place and a team of security professionals that audit the logs. The good side of having a log server is that you can correlate events over an extended pattern - for example you have smart hacker that has time to crack a root password. He will never run a open brute force attack on your system openly. He will probe your host maybe once a day from one IP address a day, but he will most likely have other hosts also so basically there can be hundreds if not thousands of attempts per day from different hosts at different times. In a singe day you may get around one or more attempts from single IP. The events will be at different times so it would look natural that some user just "accidentally" entered a wrong IP or a host name. If this happens frequently enough the logs get big enough for rotation and most likely the will rotate daily. So in 1 days time frame nothing suspicious has not happened and it is not likely that the administrators will look into the rotated logs (if they can - by default only 4 rotations are kept). This will provide an attacker the perfect cover and all he has to do is wait and hope that one of the passwords works.

Another type of attack that has become used lately is smurffed brute forcing. In this scenario a company has several servers visible to the outside world and each server or a group of servers are administered by different administrators. In this case the attacker just tries few times a day from a single IP against each server and if the communication between administrators is not working perfectly and one of the administrators is not paranoid enough to start poking in the logs it will also be unnoticed. Once one servers administrator account is compromised, installing a root kit and getting into the other servers is peace of cake.

Now, if the enterprise has set up a central log server (or a server cluster) all logs are in once place for auditing. Allso extra security is provided as the log server can be firewalled except for the logging ports so hacking the log server is harder and if a host stops login is a perfect reason to audit the server. There are several ways to achieve perfection:
log all messages into daily logs - the down side is that everything is scrambled and finding a pattern from the mess is hard
separate logs by host - better but still not perfect, as logs are still a bit of a mess and in long term grow big and slow
separate logs by host and facility - the logs will have good quality, but again the logs will grow too big and at some time have to be dumped
separate logs by host, year, month, day and facility - perfect for small to medium enterprises as you have logs that go back long time, are properly sorted and separated, new logs are created when day/month/year changes so basically it gives a good overview of the hosts status at given date also searching for a pattern from a log can be extended over several hosts
separate logs by host, year, month, day, facility and provide extra daily dumps of data from all hosts and all facilities - this method will give an extra feature not available in previous methods - the dumps can be imported into a SQL database to speed up searching and pattern matching, witch also allows the use of running previously created patterns on the logs. The drawbacks are that hard drive space is wasted on duplicates (yes keeping the duplicates is useful as one file is used for SQL dumps and other for real time auditing) also the analysis of logs can be run on dumps from yesterdays data. The storage issues can be overcome if the dumps are imported into a temporary database and afterwards moved into proper database from where there are backups also.
separate logs by host, year, month, day, facility, daily full dumps and real time database logging - this method is the mother of all. It is the most complex and has the most features available. All logs are saved into files because if the database connection is dropped nothing is lost. All logs are pushed into database where almost real time analysis can be performed with complex patterns. The daily logs can be run trough real time event correlation application that searches predefined patterns from logs and generates events on based the rules.

Witch is good for some is dependent on the needs of the company, but all work well and can are dependent on the creativity of the system administrator.

There is one dependence that the central log server needs - all the clocks have to be in sync down to a second and they have to be in the same timezone, as logs with different time are useless. One server reports that a hack happened 01:00 the other server reports that the hacked server started to hack it 06:02. Naturally it raises a question - what happened between 01:00 and 06:02? What has been done in the first server that took 5 hours? In reality the server 1 got hacked 01:00 and the hacker took only 2 minutes to root kit the server and start hacking the server 2, but with so big time difference its hard to get a real view of the events that happened.

So You have read so far and you ask - why am I blabbing about central log server in enterprise environments when I started to talk about home systems? The answer is simple - I have a central log server implemented at home also and I did some statistics today.

As I'm not a regular user I have my systems secured pretty well, but logs tell their own story. Those statistics is created purely on basis of ssh based attacks and no other services:
2004 - 29 attempts total
2005 - 71 attempts total
2006 - 194 attempts total
2007-10-24- 3794 attempts total

All that the statistics show are distributed attacks from different hosts and I don't even want to estimate what will happen next year if the attack curve stays the same. Central logging is not a magic cure against hacking but it can give powerful tools to help diagnose a lot of problems.

Blogged with Flock

LVM volume management made easy...

As a enterprise architect/infrastructure developer I'm faced with allot of problems that seem trivial on small scale but when you need to manage thousands of workstations or servers become critical problems. One of them is hard drive partitioning.

For security and stability reasons it is wise to split your root file system to several peaces - my default is:

/boot
/
/tmp
/var
/var/log
/home
swap

This may seem like crazy layout, but when you think of it, most problems start with servers getting root kited by some trivial exploit that will be used to trigger a buffer overflow and there for get a shell. If the tmp, var, var/log and home are mounted as noexec even when the exploit manages to download the overflow script - it wont be allowed to execute. This means its 1 step closer to avoid a security breach.

As I said before - it looks like a trivial task - install the server properly and thats it. Well this advice is good for up to 25..50 machines per admin, but it will fail miserably when you are running hundreds or thousands of machines. You may succeed to install them but soon after you are faced with a problem that some partitions need to be resized because one is almost empty and the other is constantly filling up.

If its just a pc you can take it offline and reinstall/boot to rescue and resize them. But when you are running mission critical servers its not an option. The same gos when you are running systems with different hardware configurations and there fore with different disk sizes. Here comes in the Logical Volume Manager (LVM). It creates another layer of abstraction on top of the physical layer therefore giving you another layer of dynamic configuration.

The greatest advantage of using LVM is that it you can create/resize/delete volumes on the fly without needing to reboot the server or boot to rescue to manage it. Sure it is not as simple as it sounds but basically it works wonders. It also gives the possibility to build the LVM volume on top of the hard drive or a raid device on full extent leaving you (in ideal case) with only 1 partition on the device (I use 2 - I like to keep /boot separately).

So in my case it is not wise make "full" partitioning of the drive, but to keep the partition sizes to minimum (that means with 10% overhead) leaving the rest of the drive free. This will allow me the simply "add" space to the volume that needs space without needing to shrink another volume (this volume cant be mounted while shrinking so this will need rescue mode!!!). Also keeping operating system on one volume group and data on another can save allot of work when restoring a system. For this I use a software designed by me called FabricManager - looks like I have to blog about in the future...

This works well until you run into a problem - You have 1000 workstations installed with this configuration and you want to do a dist upgrade. All is goo until you run out of space in the root volume to do it. 80% of the disk is filled with downloaded rpm's and there is no space left to install them. With LVM it is no problem, You just add some space to the root volume and all works out well, but here comes the catch - how long will it take to manually resize the root volume in 1000 workstations? I'm guessing around a month or two. And this is the best case scenario. Assuming that a good security policy is in place and none of the workstations have the same root password the time could be extended to few years which defies the whole point of upgrading. Sure if all the workstations have been installed with the same root password you can use an script to run the command automatically, but then You will run into about 10% of computers that wont accept the password and the 10..20% of PCs that are turned off at the time you run it. What about strict security policy that wont allow root to even log in remotely? What if the workstations are in different locations or countries or even continents? You can't just run the script few times and hope for the best as some workstations get several times the space needed and some get none.

And so a trivial problem grew out of proportions in seconds and one small design flaw can cost ALLOT of time trying to fix all the problems separately. this led me to write a small shell script that would be able to create/extend logical volumes as needed and it can be found here. It is not 100% what I need but the core functionality is there and it works.

This also solves problems with few "special" applications that need their own partition to run and as I try towards simplicity I can now package the script into the autoupdate package (right... haven't mentioned it before also... need to fix it in near future...) and set a dependence for it. So to automate the install of the application I just add one row in the %pre section of the package and woala - new volume created in the install time :D

It also helps to resize partitions by simply updating the base package of my configuration that contains the sizes of the required volumes. this gives me the ability to reconfigure workstations in few days to new configuration and life can go on...

For conclusion - security is important, so no cutbacks should be made there, noexec and nodev are your friends - use them wisely, LVM is a great tool but it is not a magic bullet - adding an extra abstraction layer multiples the complexity of the system so use it carefully and when developing large scale infrastructure multiply the the count of computers with 1000 and think if the solution is easily managed then.

This script was built for CentOS 4.4 and tested on it - it will work on other distros but may need some modifications.

technorati tags:CentOS, LVM, security, Infrastructure, bash, script, rpm, management, partitioning

Blogged with Flock

Security vs tech support...

Each system architect knows that there lies a fine line between genius and an madness the same gos for security and technical support. To protect the end users you need to trap the system down to a bare minimum and limit the users access to system resources and communication. this includes firewalls, limited package count, reduced functionality. So how do make the service desk. Around 70% of problems the technician can solve by simply login into the PC with ssh. what about the rest of the 30%? Well to not to compromise the security you cant put up a vnc server to a public net (easily spyable/fakeable passwords and user names go in plain text etc.. etc... etc...). In KDE its simple - Krfb works like a charm and can do basic digest authentication and can be configured to bind to specific interface (preferably localhost[127.0.0.1]). So to see what the user sees the user generates an support request and sends it to the technician and once the technician is finished the session is finished. This will lower the risk from technician on spying on the end user. Simple huh? Well not really - when using gnome it isn't - the gnome is so eccentric to simplicity that it lacks the configuration option, but it provides the server in essence (its called vino btw). To get it to bind to only localhost you need to hack the code, but is doable. the alternative would be to install Krfb with all of its requirements (half of the KDE environment) so it is not practical and can cause conflicts in package relations. To use the vnc you need a vncviewer package and the command to initiate the session is "vncviewr -via <the host name of the PC> localhost" - this will push the vnc data through the SSH tunnel and encrypt it on transport. Security is elemental :)

Have fun experimenting :)

technorati tags:security, vnc, ssh, help, desk, service, desk

Blogged with Flock