OSX Terminal Fun VII: Backups & History

I had a call from a frantic user today whose sendmail daemon died hardcore. This user has a habit of playig with things first and then getting into trouble when the whole thing breaks- they aren't technically inept, they can follow instructions... and they'll remain nameless as they know who they are, and they owe me some pie now. :)

...but where I'm more of a "Learn everything about what I'm about to do before I do it" they're more of a "Play with it and when it breaks learn from that" type of person.

They weren't really amused by the problem- with sendmail dead on their box, they not only couldn't send mail from their accounts, none of their scripts (for support, sales leads, etc) were outputting either. To say they wanted it up as fast as possible was an understatement, unfortunately they weren't completely sure of what they did to cause the problem, just that they had been trying to edit the sendmail config files for better use of the 5th element for their PHP scripts, which they'd been reading from some site.

If you are playing around with the guts of any OS (linux, bsd, osx, windows, whatever) you need to have wholesale backups. But since people get told that all the time and they still don't do it, at the very least create local backups of the files you change.

Fixing the problem wasn't that hard, but first I needed to know exactly what they'd done to cause the problem: ie, the affected config files. I did this with the history command. It was pretty easy to figure out what they'd done when I saw:

pico /home/virtual/site/fst/etc/mail/sendmail.mc

and

m4 /etc/mail/sendmail.mc > /etc/sendmail.cf

You can look up what the last command does, but suffice to say it did a big nasty. But by knowing what they'd done I was able to replace the various files that were affected. If this person had followed a few rules, it wouldn't have really been an issue. Since I've already gotten mail from some of you trying the commands I've given and botching them, here are a few more unix 101's:

history
who & last
doing basic backups of a file before you modify it.

The history command

Any unix system will log a users actions into a history file, the name of which is different depending on your shell. You can change how much it remembers, or even have it set to nothing at all. You can either view the file directly, or use the following commands (or variants of).

% history

Will show you all of the commands logged into the history file, chronologically and numbered. This is per-user, ie if you are logged in as root it will show the commands executed by root, if you are logged in as an admin it will show those. What it won't do is show commands executed from within a an application. For example, you might see:

890 pico -w /etc/sendmail.mc

But it won't show you what you did to the file while you were editing it within pico. It's very valuable though, as it can show you what files were touched. Other variants that are helpful are:

% history | tail

...which pipes the history output to the tail application and shows you the last 10 commands executed. You can "% man tail" for more info, but the -n modifier is usually the most useful. Ie:

% history | tail -n 20

...will show you the last 20 commands executed by the user.

% history | more

...will display all of the commands stored, but will pipe them to more to only see one screen at a time.

For bash, history files are usually stored as .bash_history within the user home account, for tcsh they are usually stored in .tcsh_history. You want to know this because if you want to know what a user has done, you might do something like:

% pico -w /home/username/.bash_history

...for bash shell users, or:

% pico -w /Users/username/.tcsh_history

...for tcsh shell users.

Who and Last

Very basic. Whenever someone logs into a unix system, it's logged into a file.

% who

...will display who is logged into the system currently.

% last

...will give you a list of who has logged into the system over time, as well as when.

% last | head

...will give you the last 10 users who have logged into the system, you'll see your current session listed as "still logged in".

Beyond the security implications, these are important because if you know about when something when wrong, the above commands can help you figure out who may have done it. In other words, if a site starts going wonky at 2am, you might do:

% last | head -n 30

Where you see that user drunkenbatman was logged into your system at 1:30am and logged off at 2:30am. Chances are they did something you want to check out, so you would:

% pico -w /Users/drunkenbatman/.tcsh_history | head -n 20

...which would give you the last 20 commands that user executed, and can set you on the trail for figuring out what they did.

Doing basic backups of a file before you modify it

This is extremely basic, but it just doesn't occur to some people. If I was going to modify my ~/.bashrc file in order to add aliases, before you do anything you should first do something like:

% cp ~/.bashrc ~/bashrc.bak

...which does nothing more than create a backup of the file in the same directory. You could call it .backup, or .modified, but whatever you do pick something you will stick with every time you do it. For instance, you might want to know exactly what files you have specifically modified, and if you have been good about using the naming convention .bak you could use the command:

% locate .bak

...and the system would return the name and path to every file located on the system with that extension. If you see something like:

/etc/httpd/httpd.conf.bak

...you'll know you have at some point edited Apache's /etc/httpd/httpd.conf.

You have to do something like this if you are going to be playing with a unix's guts. No matter how careful you are, and or how sure you will be wanting to keep the modified file, you need to do it. Another trick is that if multiple users will be doing the above, they can each use a different suffix for their backups.

The sendmail daemon issue I had to fix today on the users machine could have been fixed in less than 5 minutes if they had done the above, instead of having to copy everything over from a virgin machine.