Category Archives: Uncategorized

Ammonoidea

It is pretty typical of servers to be without monitors or even without graphics cards. In the process of deciding on some new Sun servers and their configuration, we were discussing terminal servers to allow easier access to the serial consoles. We had several options we were considering. Being Macaholics, we chose the option of using a cast off iMac with Mac OS X as a terminal server. We had already done this sort of thing with my desktop G5 and with our shared MacBook. So we knew how to do it, how to secure it, and could get it done quickly and easily. The only expense would be a USB to Serial Port adapter. We have been using Keyspan adapters and chose the 4 port adapter which we found new on Amazon for $136 with free shipping.

Sorting through the available iMacs, I found one of the last of the CRT iMacs that was made. It is a 700MHz G4 with 256MB RAM and a 60G drive. Well adequate for our needs. The details on it and the latest OS that could be installed on it were found at everymac. So the first step was to wipe the drive and install Mac OS X 10.4, followed by a sequence of Software Updates that ended with Mac OS X 10.4.11. Since the CRT iMac is an “obsolete, shelled creature” that is going to have multiple tentacles (serial cables), I named it Ammonoidea. I made an entry for it in our DHCP server and left the network to configure automatically (actually did this before doing the install and updates).

Mac OS X comes pretty well locked down to begin with. Initially, I left the firewall closed off and all services off. I set the screen saver to require a password, locked System Preferences, and set a few other cosmetic options. At the end of the setup, I opened SSH (Remote Login in the Sharing Preferences) and controlled access to it with TCP-Wrappers. Since SSH is built with the wrappers library, this last step just involved setting up /etc/hosts.deny (ALL:ALL) and /etc/hosts.allow (sshd:<list of allowed IPs>). See `man tcpd` and `man hosts_access` for reference. More complete information on securing Mac OS X Tiger can be found among the NSA’s Security Guides.

The configuration of this system was fairly straightforward. It uses the terminal application to get a command line window and GNU Screen to create detachable serial console sessions. It turns out that Screen is included in Mac OS X. So, with the addition of the Keyspan adapter and the drivers, the iMac with Mac OS X is practically a terminal server out of the box. (Oh, I also grabbed Firefox, just because I always prefer using Firefox). To complete the setup, I needed to configure user accounts, configure screen, and configure a startup service so that the detached screen console sessions would always be there and always be logging.

I decided to set up a sysadmin account for the administrator of the systems whose consoles we would be connecting. In Mac terms, this was set up as an ordinary account without administrator privileges. This is the account that screen will be running as and which will be logged in at the physical interface of the iMac. Then, each administrator has his own account with administrator privileges. We can ssh into those accounts from a limited set of IP addresses within our private network (controlled by TCP Wrappers). Using screen, we can then access the console sessions remotely using, e.g., `screen -rx sysadmin/nemo`, where the “-r” is reconnect, the “-x” is multi-user, and “nemo” is the name of the console session shared by user “sysadmin”.

Screen is a bit of a swiss army knife. `man screen` gives a lot of details, and is important as a reference, but it isn’t exactly an introduction or guide. I found a basic introduction and tutorial at linuxhacks, and a bunch of tips at Softpanarama. I also found the man pages online and a number of links on Wikipedia. Having all of these tabbed in Firefox provided an easy reference during setup. Essentially, aside from learning the ins and outs of screen, all that was needed was to figure out a proper setup for the ~sysadmin/.screenrc configuration file. What worked for me was:


startup_message off
msgminwait 20
msgwait 30
multiuser on
acladd wallace,gromit,sysadmin
aclchg wallace,gromit,sysadmin +rwx "#?"
chdir
autodetach on
logfile log/screenlog.%t
deflog on
logtstamp on
defscrollback 100000

The group of commands that sets multiuser on, adds the users, and changes their accesses to full allows me to run detached sessions that can be reattached by any of those users, even when others already have them attached. This means that the console session for server “nemo” can be on the physical interface of the iMac where user “sysadmin” is logged in while either (or both) wallace and gromit can ssh in and attach the same console session. If one administrator types something, anyone else with the session attached will see the typing and the resulting output. To make multiuser work, I also had to make the screen binary SUID root (`sudo chmod u+s /usr/bin/screen`).

The last group of 4 commands sets up the logging. I created a directory ~sysadmin/log where screen logs are kept. The logs are named screenlog.%t where %t is the title specified when the screen session is started. That allows me to title the sessions for the servers that are on the particular ports. By default, screen will append logs to an existing file. So I will end up with an ongoing log. The deflog sets logging on by default, and the logtstamp puts a timestamp in session (and thus log) when it has been inactive for 2 minutes. It will put a matching timestamp in when the session becomes active again. The logging should go on indefinitely, limited only by disk space, but the defscrollback esures that I can scroll back in the screen session without hitting a blank wall.

Some of the rest of it is more cosmetic.

Finally, I want screen to automatically start when the iMac is turned on, not when the user sysadmin logs in. I found details on how to do this in the O’Reilly book “Mac OS X Tiger for Unix Geeks” starting on page 60. I found even more detailed information in the Apple Developer Documentation. While, technically, Apple has moved away from startup items and uses launchd starting with Tiger, they still support it for user and 3rd party applications for backward compatibility. I chose to do it this way because it was simpler. Basically, I created a directory /Library/StartupItems/Screen. Inside that directory, I created an executable shell script “Screen” and a property list “StartupParameters.plist”.

The plist file is:


{
    Description = "GNU Screen detached serial consoles";
    Provides = ("ScreenConsoleSessions");
    OrderPreference = "Last";
    Messages =
    {
        start = "Starting GNU Screen";
        stop = "Stopping GNU Screen";
    };
}
   

The shell script is:


#!/bin/sh
# based in part on Apple developer documentation and
# in part on the example in O'Reilly's "MacOSX Tiger for Unix Geeks"
   
# Source common setup, including hostconfig.
#
. /etc/rc.common
   
# The start subroutine
StartService() {
    # Insert your start command below. For example
    # mydaemon -e -i -e -i -o
    # end example.
   
    # Don't start unless screen is enabled in /etc/hostconfig
    if [ "${SCREEN:=-NO-}" = "-YES-" ]; then
        # ConsoleMessage "Starting GNU Screen detached serial consoles"
        # a plain screen session and 4 console connections
        su - sysadmin -c "screen -dmS base -t base"
        su - sysadmin -c "screen -fn -dmS nemo -t nemo /dev/tty.USA49W191P1.1"
        su - sysadmin -c "screen -fn -dmS dory -t dory /dev/tty.USA49W191P2.2"
        su - sysadmin -c "screen -fn -dmS gill -t gill /dev/tty.USA49W191P3.3"
        su - sysadmin -c "screen -fn -dmS jacques -t jacques /dev/tty.USA49W191P4.4"
    fi
   
}
   
# The stop subroutine
StopService() {
    # Insert your stop command(s) below. For example
    # killall -TERM mydaemon
    # sleep 10
    # killall -9 mydaemon
    # end example.
   
    # each instance has a root process SCREEN and user process screen
    # killing the root process kills both
    killall SCREEN
    sleep 10
    killall -9 SCREEN
   
}
   
# The restart subroutine
RestartService() {
    # Insert your start command below. For example
    # killall -HUP mydaemon
    # end example.
   
    # Don't start unless screen is enabled in /etc/hostconfig
    if [ "${SCREEN:=-NO-}" = "-YES-" ]; then
        # for Screen, HUP doesn't make sense.
        StopService
        StartService
    else
        StopService
    fi
   
}
   
RunService "$1"
   
   

Now, after adding “SCREEN=-YES-” to /etc/hostconfig, if I reboot the system, I find that it starts up with all the console connections as detached screen sessions with output going to the log files. The sessions are under the user sysadmin. Each session is associated with a root process “SCREEN” and a sysadmin user process “screen”. The stop portion of the shell script only has to kill the root process to stop both. The lengthy port names, e.g. “/dev/tty.USA49W191P1.1”, are set up by the KeySpan drivers.

If I ssh to ammonoidea as wallace, I can see all the sessions by doing `sudo su – sysadman -c “screen -list”`. But, since they will always be the same, and should always be there, all I really need to do is `screen -rx sysadmin/nemo` and I will be connected to the console session for server nemo. Within screen, `Ctl-a` is the escape character for screen commands. `Ctl-a d` will detach the session. I could get along with nothing more than that. However, I’ll probably need to be able to scroll back. `Ctl-a [` puts me in the scroll-back and copy/paste mode. I can use the same keystrokes as vi to scroll back in the console session. So `k` is back, and `j` is forward. Left and right don’t really matter in this setting. While in scroll-back mode, `Ctl-u` will scroll back half a screen-full, and `Ctl-d` will scroll down half a screen-full. `Esc` gets out of scrollback mode.

If at any point I needed to manually start or restart the detached screen sessions for the consoles, I can do it with Apple’s command line utility “SystemStarter” by, e.g., `sudo SystemStarter restart ScreenConsoleSessions`. That last item in the command is the service name I defined in the plist file for the Screen startup item.

That’s basically it — an easy to use, secure, BSD based console server that’s inexpensive and totally under our control for configuration and updating. Not an expensive black box. Of course, you could do all this with generic PC hardware and your choice of a linux or BSD distribution. KeySpan makes 4 port PCI serial adapters. But, we’re Macaholics, and we thought this was easier and more secure out-of-the-box without a lot of work.

Ten Things I Like about Amanda

Nothing kinky here. Amanda is not a girl, but rather a really nice piece of software —
The Advanced Maryland Automatic Network Disk Archiver. In short, a backup program. Details can be found at http://www.amanda.org, or at http://www.zmanda.com (with documentation at http://wiki.zmanda.com). I’ve been following Amanda for a couple of years, and started using it intensively last January. I’ve been extremely happy with my decision to implement Amanda, so I decided to list some of the reasons.

Digg this post.

1. Amanda is a simple, elegant and powerful tool. It is a planner and scheduler for backups. It does what it does better than any other program, and it doesn’t try to reinvent the wheel or throw in the kitchen sink (see 2 & 3). There is depth, maturity and complexity to Amanda, because it has been around for quite a while (over 15 years) and the task it undertakes has some complexity to it. But it has stayed focused and true to its original, fundamentally sound design concepts.

2. Amanda uses native tools to do the actual backups rather than trying to write its own. This has a variety of implications. (a) It is using tools that have a broader base and more extensive testing than would be possible if Amanda developers wrote their own backup code. (b) It gives the user a choice. For example, on Solaris you can use ufsdump, and on Linux you can use gnutar. (c) It avoids a whole class of potential bugs that could creep into the program (gee, we didn’t get those acl’s on Mac OS X quite right). (d) It makes it relatively straightforward to extract and recover data even if Amanda is unavailable to assist. (e) It makes it easier to support a broader selection of platforms.

3. Amanda is minimilist in that it doesn’t require the installation of other significant packages (beyond what’s typically on a Unix or Linux system by default) for functionality that is critical to Amanda’s operation. For example, it doesn’t store records and indexes of its backups in an SQL database. So, you don’t need to get anything else running to run Amanda, you don’t have the system load associated with other software (or need an additional server to run it), and you aren’t cripled in a disaster recovery situation by having to get more stuff up and running before you have full functionality.

4. Amanda is really good at multitasking. During my testing, I had terminal sessions open to several of my servers. They each had top running with the command to display jobs for user amanda. When I launched amdump on the server, I almost immediately saw multiple jobs firing up on all the servers. Amanda had estimates going on all of them, and soon had data streaming in from several of them. I had two spool drives, and I could watch data accumulating on them and then going out to tape. With adequate resources and tuning parameters, Amanda can orchestrate quite a show.

5. Amanda’s algorithm for scheduling backups is awesome; and, as far as I know, there is nothing like it in any other backup software. If you’ve been doing backups with other software for years, it might be a little difficult to get your head around this, but it really feels good when you do. Basically, you choose a dump cycle, say, a week, and runs per cycle, say, 5. Amanda guarantees that you will have at least one full backup of every disk list entry during a dump cycle. On the other days it will schedule incrementals. The distribution of fulls and incrementals across the dump cycle is planned so that the amount of data being transferred and put on tape is about the same every day. The benefits of smoothed out network load, smoothed out demand on other servers, and even use of tape from day to day is tremendous. And there really is no downside. You’ve always got incrementals, so you can always recover to any day. No worries. Just let Amanda plan it.

In a more general sense, Amanda’s backup levels cover the full range from 0 to 9. In common language, a level 0 backup is a full backup, a level 1 backup is what some people call a differential backup, and anything from level 1 to 9 can be called an incremental. A backup at any level will backup all the files under the disk list entry that have been modified since the last backup at a lower level. Amanda chooses the backup level to optimize the balance between backup efficiency and ease of recovery. And all of this is tunable by configuration parameters.

6. Amanda is extremely robust against error situations. It typically does the right thing, and administrator intervention is only required when it really makes sense. For example, I had a tape drive failure several months ago. For a variety of reasons, it was out of service for a couple of days. My attention was focused on the tape drive, and I really didn’t want to have to think about a lot of other things. Amanda was scheduled to run daily backups. At the scheduled time, Amanda “surveyed the situation” and made an operational decision. No tape drive is available. Backups must be run. Holding disk space is available. Holding disk space should be used conservatively, since it can’t be flushed to tape. So Amanda dropped back and did incremental backups on all the disk list entries and saved it all to the holding disk. It sent me an appropriate status report with a notice about the tape failure at the top. This went on for a couple of days. Then, when I got the drive repaired and reinstalled, Amanda said, “Oh Joy”, scheduled an appropriate mix of fulls and incrementals, and flushed everything out to tape. I never had to say anything to Amanda during this episode that spanned a couple of days. It just worked.

7. Amanda is a collection of command line utilities. You configure Amanda in part by making cron entries to run these utilities (amcheck and amdump). When no backup is running, there are no processes or daemons on the system, and there is no need to monitor the system to make sure the daemon is still running. Typically, cron is configured to run amcheck during the afternoon when sysadmins are around. It checks everything and notifies sysadmins only if there is a problem that needs attention. Cron is configured to run amdump after the end of the day or in the middle of the night. This follows on item 2 by using the native system scheduling tools (cron) combined with Amanda’s backup scheduling algorithm. On the client, Amanda is triggered by inetd (or xinetd), so, again, it is only running when a backup is happening and a request comes in from the Amanda server.

8. It is extremely easy to get appropriate information about what is going on with Amanda. Typically, a routine amcheck is scheduled in the afternoon to check the readiness for daily bakckups. It checks the configuration, the backup media, the connections to the clients, and only reports if there is a problem. Because amcheck is a command line utility, you can run it at any time. Whenever amdump is run, it sends a report with its results. In addition, if you come in in the morning and haven’t gotten the daily amanda backup report yet, you can run amstatus and find out exactly what it is doing. There is also amreport, which is a versatile reporting tool for accessing information about existing backups and tapes.

9. The community support for Amanda is very stong. Open source projects typically rely on community support, but the results are highly variable. Amanda falls into the group of projects that have exceptionally good support. The activity on the Amanda users list shows a large number of respondents answering questions on a variety of platforms; and, according to Preston*, over 250 programmers have contributed to the Amanda code base.

10. Amanda has commercial support through Zmanda. Normally, managers have to choose (typically, sysadmins only get to recommend) between open source or commercial software, and can’t balance the virtues of the two. Significant open source project often have a community of consultants and freelancers who can be hired; but, with Amanda, there’s also commercial scale support, if that’s what a manager wants, with up to 24/7 and unlimited support calls. And, because Zmanda strongly supports the open source model, and has paid programmers on full time staff, the overall level of development and support for Amanda is richer. The open source community has access to this through the participation of the Zmanda staff in the user community and their contribution of code.

*Preston, W. Curtis. 2006. Backup & Recovery. O’Reilly, Sebastopol, CA. http://www.oreilly.com/catalog/9780596102463/).

The Ultimate Minimalist Monitoring Tool

I spent a lot of searching for and looking at different monitoring tools. There are a lot of them out there — both open source and commercial. The basic idea is to automate the process of keeping an eye on your servers or network. You want to know if the server is up, if the services it is supposed to provide are available, and so on. The variation among the tools is in the details. What language is it in, how is it configured, how is the checking done, how does it alert you, how can you check up on it, does it have a GUI, does it produce graphs, and so on.

I had somewhat reluctantly zeroed in on Hobbit Monitor, which is an offshoot of Big Brother, but seemed to be more focused and together. It is programmed in C, whereas BB is shell script and Big Sister is perl. They all assume a web interface using Apache and use RRDTool to generate graphics. They also typically assume a client application installed on the server that is to be monitored.

My idea was to install monitoring software on my backup server which is running Amanda and doesn’t have much else on it. I didn’t really like the idea of adding more packages and opening up more ports just to get a monitoring application running, and all the alternatives I looked at seemed to have too many requirements. Some would use Java. Some wanted PHP. It just seemed like escalating software infrastructure. Zenoss wanted Python, a web interface with Zope and Ajax, Zope ODB, MySQL, RRDTool, . . . aaarrrggg!

Finally, I stumbled on mon. Try googling that. It’s virtually impossible. Just remember that it’s at kernel.org. Mon is my dream tool. It is simply a scheduler of monitors and alerts. It is written in C and takes a typical text based config file. The monitors and alerts are separate scripts that can be in any language. Many of them are in perl. You can collect whichever ones interest you, and you can write your own. Mon’s footprint on your server is so small that it is hard to see. It doesn’t eat CPU. It doesn’t eat memory. It’s almost invisible … until you get an alert.

There are a wide variety of monitors, including fping, dns, http, smtp, mySQL, mSQL, PostgreSQL, ftp, and dozens of others. Typically, they do more than just poke a port. The mySQL monitor, for example, tries to login and list the tables. There are also a number of alerts, including mail, irc, pager, and a few others. I have ambitions of taking the alert template and writing a growl alert [“Mail’s down. Quick, send an email alert! Oops. Umm, Growl?”] Growl is an interface to the alert system on Mac OS X. Remote Growl opens a port to receive remote alerts. So, mon could pop an alert up on my Mac desktop. All I need is the interface script and then add it to the config file.

You can find mon at http://mon.wiki.kernel.org/. Growl is at http://growl.info/. Remote Growl is at http://remotegrowl.erlang.no/. Other things mentioned above, you either don’t need or can find easily enough through google.

What is a Minimalist SysAdmin?

Short version

The opposite of the bloated, kitchen sink, Windows administrator.

Long version

A Minimalist SysAdmin is one who defines his needs in terms of necessary functionality and tries to choose the simplest solution with the least number of pieces that serves those needs. An ideal solution will be simple, elegant and powerful.

One of the paradigms of Unix is the tool that does a particular task well. Such tools can then be used in tandem or sequence to solve more complex problems. The essential Unix toolbox contains a number of such tools. Typically, they are designed so that they can take their input from standard input and send their output to standard output. This means they can be connected together in a piped sequence. For example, a typical Unixy sort of thing would be to do something like

$ cat /var/milter-greylist/greylist.db | awk ‘{ print $1 }’ | sort | uniq | wc -l

which takes a text database, grabs the first entry on each line, sorts them, removes duplicates, and finally counts the number of remaining lines. Since the first entry on each line in greylist.db is the incoming IP address, this gives the number of unique IP addresses represented in the database.

Besides the simplicity and power of this approach, there are other important concerns, including security, performance and maintenance.

Security is always an important issue. The more features, connections and angles a piece of software has, the more likely it is to have a bug that can be exploited. The process of installing and securing a system includes a significant portion of time spent eliminating unnecessary or risky functionality. In its simplest form, this means removing things like rsh, rlogin, telnet and ftp and requiring the use of ssh or scp. In a broader sense, it develops into an approach used to make choices. Given two software packages that serve the same purpose, choose the one that solves the problem simply and elegantly. Don’t choose the one that requires the installation of several additional packages in order to achieve a web GUI administration interface with animated charts. You probably don’t need that functionality, and it opens you up to many more possible exploits that could compromise your system.

Performance concerns have some things in common with security concerns. A simple elegant solution may have a very small footprint on your system in terms of resource demand. Add all the extra stuff and you may find your system bogging down. That may make the manufacturers of memory and CPU upgrades happy, but it won’t make you happy unless you like spending more money on hardware gadgets and upgrades all the time while getting less out of your existing machine.

Maintenance is easier when you don’t have all those other packages to be concerned about. Every piece of software installed should be tracked for security vulnerabilities and updates. When an update is required, compatibility with other interdependent packages on your system has to be taken into consideration. A complex installation which requires one piece to be upgraded may cause a cascade of necessary upgrades that have to be done in the right order. In the worst, this may require that important services be down for a lengthy period while upgrading. Or, if something fails, the troubleshooting can be much more difficult and the resultant downtime longer.