In Red Hat/Fedora land, Yum now reigns fairly supreme for package management. But Yum is still just a front end for RPM (Red Hat Package Manager), and you still need to know something about the lower-level tool.

The utterly crucial thing that RPM does for you is encode the dependencies of one package on others. (The utterly crucial thing that Yum does for you is resolve those dependencies.)

A subtle point is that dependencies are expressed in terms of capabilities. (You will search in vain to find this term carefully defined in RPM land.) It seems to be: "A string of letters someone invented to describe... well... a capability."

The Provides: lines in an RPM spec file specify capabilities, as do any Requires: lines.

It makes life easier (for spec-file writers) that every package implicitly provides a capability <epoch>:<packagename>-<packageversion>.

How will RPM capabilities turn up in your life?

Well, when you find nifty RPM foo-1.2.3.fc10.i386.rpm and try

yum localinstall foo-1.2.3.fc10.i386.rpm
# NB: the yum equivalent of ...
#  rpm -Uvh foo-1.2.3.fc10.i386.rpm
# ... except dumber.

and it complains loudly, it will complain in terms of capabilities.

So when it says it lacks /lib/libfoo.so.0, it isn't complaining about the missing file per se -- it is complaining that it is missing a package that provides the capability named /lib/libfoo.so.0.

Knowing that, the commands to investigate capabilities may get you to the answer more quickly:

• What capabilities does a package provide?

  rpm -q --provides <pkg>
  

• What packages require one of those capabilities?

  rpm -q --whatrequires <capability>
  

• What package provides a capability?

  rpm -q --whatprovides <capability>
  

[An earlier version of this note appeared in Verilab's internal newsletter.]

I am a fan of SSH (Secure Shell), which moves bits around safely in a myriad of amazing ways. Tunneling, X11 forwarding, keepalives, passwordless access, ... it just gets better and better.

One part I didn't get my head around until recently is the "passwordless" thing and exactly how far you can go with that.

First there were keys...

In the simplest case, you type ssh ssh.example.com, give a password, and you're in.

An alternative -- and potentially passwordless -- way to make an SSH connection is with "keys" (i.e. using public-key cryptography).

For this, you have a pair of keys, the "private" one on the client and the "public" one on the server. (Conceptually) The SSH client encrypts something with the private key and, if it decrypts successfully with the public key, then it must have come from a Good Guy, and you're in.

You can make a passwordless key pair by typing:

ssh-keygen -t rsa -N ''

Copy the public key that's generated over to the server and put it in ~/.ssh/authorized_keys and you're off. (Actually, it's very fussy about permissions and things; never mind, there are hundreds of how-tos about that all over the 'net.)

The security problem here is that, if someone gets at your client account (as you, as root, by nicking your laptop, ...), they've got your private key and can use it.

The first step back from the brink is to put a "passphrase" (a password in high heels) on your private key. Leave off the -N argument to ssh-keygen. (You can also use ssh-keygen to add a passphrase to a previously-passwordless key.)

But, strangely, we're back where we started: Giving a passphrase every time you use a key feels exactly the same as giving a password each time.

... And then there were agents

The tool you want (and the main point of this note) is ssh-agent.

It is a program that starts up, you tell it about your keys (i.e. hand over the passphrases), and then the SSH client cooperates with the agent to hand out keys when required -- no passphrases.

So: machine boots up, you login... quick session with ssh-agent... no passphrases after that. Perfect.

Nowadays an ssh-agent is started for you as part of the login process. Do env | grep SSH_, and if you see something for the environment variable SSH_AUTH_SOCK, then you are already agent'd up.

How do you tell the agent about your keys (or "identities" as it likes to call them)? That's what ssh-add is for. If you have just one key (not unusual), then just type:

ssh-add

give your private key's passphrase, and declare victory.

You can list keys/identities managed by the agent (ssh-add -l), or delete a key (ssh-add -d), or lock your agent with a password while you go to lunch (ssh-add -x).

How does the SSH client know which ssh-agent to chat with about your keys? It gets the information from those environment variables SSH_AUTH_SOCK and SSH_AGENT_PID.

And what about cron jobs? (or other scripted tasks)

A hard passwordy problem is cron jobs, with something like backups being a common and nasty example. You need to move bits from one machine to another (so SSH is just the thing) but you need to do it as root. It must run completely unattended.

Putting the root password in a backup script -- no doubt often done... -- is a complete non-starter.

Passphrase-less keys are much better, and OpenSSH provides many extra ways to mitigate the risk, e.g. only allowing connections from certain IP addresses.

But the fact remains: If someone gets one of root's passphraseless private keys, it's fairly worrying.

So can we play the ssh-agent game to limit this risk? Yes.

It is possible to start up an agent by hand (or by script), and save the SSH_ environment variables (which it will emit) in a Well Known Location. Scripts can then source those variables (from the Well Known Location) and get the benefit of the agent.

Is that the best we can do?

Yes. When a machine (re)boots, an ssh-agent can be started, the needed environment-variables recorded, and someone ssh-adds all of the keys that will be needed (i.e. gives their passphrases). That is the least human interaction possible for really quite-secure unattended SSHing.

As long as the machine stays up, scripts should be able to work without any passphrases.

Lingering risks

The SSH_AUTH_SOCK environment variable points to a socket in the filesystem (with restricted permissions). Surely, if I can get read/write permission to that socket (e.g. if I'm root), then I can use the ssh-agent to take advantage of post-passphrase SSH keys?

Yes, that's right. Two things: (a) If someone is already on your system, perhaps as root, you already have a big problem, never mind SSHing. (b) The bad guy can use your key through the agent, but cannot get a copy of the private key itself.

Summary points

There are almost-as-convenient ways to use passphrase-laden SSH keys as the much-more-vulnerable passphrase-less ones. Use them!

You can get comparable security even for unattended automated tasks (scripting).

[An earlier version of this note appeared in Verilab's internal newsletter.]

I wanted to change the RSS feed for this (MovableType) blog over to FeedBurner. You might think this would be a routine configuration change.

You would be howlingly wrong. After logging in, pick Design -> Widgets. (Is there a more meaningless word than "widget"?)

Now scroll down to and click on Syndication. (Yes, intuitively obvious to the most casual observer.) No, don't tick the box and look for something to try under More actions... -- your only choices there are Refresh Template(s) or Clone Template(s) (whatever those are).

OK, you clicked. What you now see in front of you is a VIM-like text box of pseudo-HTML:

<div class="widget-syndication widget">
    <div class="widget-content">
        <ul>
            <li><img src="<$mt:StaticWebPath$>images/stat...
<mt:If name="search_results">
    <mt:If tag="SearchString">
            <li class="search-results-feed"><img src="<$m...
    </mt:If>
</mt:If>
        </ul>
    </div>
</div>

Just make the right change to that and you can use FeedBurner as your feed. Am I the only one having a laugh-or-cry moment?

(Take a deep breath. This is a long screed, mostly because I didn't find something else that laid out theory/practice to my satisfaction.)

Everybody hates backups. But the cold hard fact is that every computer's life ends in tears. A disk drive will fail. Coffee will be spilled on electronics. Laptops will be left on trains.

It is less pain -- much less pain -- to set up something for backups in the first place. This article gives both the "theory" and also a nuts-and-bolts guide on how you might proceed. (NB: I have not tested all of this stuff.)

This article is XP-oriented. Vista may be slightly different/better; I don't know.

Why Windows is horrible for backups

The more you learn about backing up Windows computers, the more you will think it was malevolently designed. There are several root causes.

First, system and user data and configuration are thoroughly commingled on your C: drive. (Many smart people say the first thing you should do with a Windows box is create a D: partition and arrange that user stuff ends up there. But that barn door is open and the horse is gone...) It is hard to mess with the one but not the other. It makes every backup task turn into multi-gigabyte shifting, even if you just wish to preserve a few megabytes of letters from your granny.

Second, the registry, which is a system database of settings: it is impenetrable, delicate, and interwoven with every part of the system. You can't restore an application's files and then be up-and-going again... you need the relevant registry settings, too. But a mere mortal can't get at those independently...

Third, Windows "tailors" itself to your particular hardware. So you can't take a working hard drive, shove it in another machine and get going again. Linux systems (for instance) can do exactly that. It's just Microsoft intentionally screwing you. (When your motherboard goes 'poof!' and you're stuck, I'm sure you are delighted to know that you are helping Microsoft with its anti-piracy efforts.)

Backup desiderata

Let's first stop and think what we want from a backup system. It's a longer list than you might think.

• Be able to restore a machine to "just like it was", whether that is a week ago, a day ago, an hour ago...

• Be able to restore from "bare metal" (for example, if the old C: disk drive went to the rack enclosure in the sky). (Aside: the tool for this is often a Linux live CD dressed up in high heels.)

• Be able to restore one machine to "just like another one was". Laptop left on train? Get another one, splat on a backup from yesterday, and you're exactly where you were.

  Cloning tools, used for fleets of machines, work for this task, too.

• Be able to restore when away from base, e.g. when the train you left the laptop on is in another country.

• Be able to backup/restore files that require special pleading, e.g. database files, or the registry, or maybe even your Outlook PST file.

• Be able to take backups without putting the machine into some special "you can't get work done" mode.

• Be able to recover the file (or folder) that you accidentally deleted. This goal typically requires some user-friendly GUI with which to view the backed-up material and pick the item(s) to be recovered.

  Ideally, this facility would let you get at any file that was ever opened/closed. More likely: frequent (e.g. hourly) snapshots -- and you can have any file alive at the time of a snapshot.

• Have the backup system work without user intervention. For example, if a backup disk fills up (as it inevitably must), the system should quietly-and-correctly toss the oldest stuff and keep going.

• Something that will indicate that the backup system hasn't just died. You don't want to find out that your fully-automatic backup system that you started two years ago has, in fact, been failing hourly for a year and a half. At least some tools allow notification by e-mail or IM, as well as some "verify this backup" facility.

• Can battle on in the face of disk errors... Say you get some disk error(s), so you want to take that "final copy" of an old drive (from which you will restore subsequently). You don't want a tool that stops at the first whiff of trouble.

• Ability to put the backups on various media: DVDs, hard drives (internal or external), a "network drive" (e.g. Samba), some funky out-there-on-the-'net thing, ... Err.. and restore back from 'em.

• Encryption of the backups: No point having a secure laptop, if a copy of its backup is free and easy...

• Backing up is a kissing cousin of synchronizing data on two computers. A tool may serve both purposes.

Principles of Windows backups

The main thing to know about Windows backups is that there are two major kinds and they do different things.

An image backup is a bit-for-bit copy of the filesystem. (A 'sector-by-sector' image backup is a block-by-block copy of a partition, the same thing you get with 'dd if=/dev/sda1 ...' on Unix.) An image backup is usually smart enough to record only the part of the disk (partition) that is used, and it is usually compressed, often up to something like 45%.

Here's the main point: You must have an image backup if you want to put a Windows machine back just like it was.

A file backup (my terminology) is what it sounds like, a copy of a bunch of files-'n'-folders in the filesystem. A common example might be My Documents.

Note well: a file backup is portable from machine to machine. An image backup can (for most practical purposes) only be used on the machine it was taken from, perhaps with a new hard drive.

You can have a full backup of either the image or file flavors. (Again, you can only make a machine "like it was" with a full image backup.)

You may also have an incremental backup of either flavor. These come in two forms:

• A differential backup is always against a full backup. If you take a full backup on day one, then differentials on days two and three, the differential on day three will also include the changes made on day two.

  You would only need the day-one full backup and the day-three differential backup to do a full restore.

• An incremental backup (we will use this narrow sense of "incremental" from now on) is against a full -- or the previous incremental. If you take a full on day one, then incrementals on days two and three, the incremental on day three only includes the changes between days two and three.

  You would need the full and all subsequent incrementals to do a full restore. But incrementals take less space than differentials.

Making the choice

• No image backup, with file backups:

  This is the "I don't mind doing Windows installs" option (including re-installing all the applications that you use). In this case, being able to reach out and get your old data files (from file backups) should be fine.

• Image backups, but no file backups:

  This is good for doing bare-metal restores. Also, at least with decent software, you should be able to pluck individual files for recovery from such images. (NB: Worth checking... you'll not be happy when, to get back yesterday's letter from Granny, you have to re-image your whole C: drive.)

  The downside is that the differential/incremental images take up way more room than for file backups.

  This choice may be suboptimal for the laptop-on-train case. You probably won't be able to use the image to get a (new) machine "like it was". But the software may let you use the image backup as a source of recoverable files/folders.

• Both image and file backups:

  In this scenario, you take the occasional image backup for "restore-from-bare-metal" purposes, and use some file-backup system for day-to-day letters-from-Granny purposes.

• No image, no file backups:

  The "I don't need backups" option. If you do all work "in the cloud" (e.g. Google Mail and Docs) and don't mind doing an OS reinstall if it comes to that, then you're cool.

"Both image and file backups" is the right answer for most people.

What I would want is: a works-automagically ("unattended") backup system that does:

• A full image dump once a month;

• A differential image backup once a week;

• A "My Documents"-ish (folder) dump once a week;

• An differential on that once a day.

• Re-uses space sensibly.

• Sends me confirmatory email once a day.

Vendor backups

The vendor of your laptop (or other computer) is likely to have supplied some "recovery" mechanism with which you can get the machine back to a known state. These mechanisms are opaque, partly because the vendor doesn't tell you exactly what it does and partly because no sane person looks into the matter until it's too late :-(

We will lump these various mechanisms under the label "vendor backups".

One common wheeze is to have a "secret" partition on the hard disk with a separate copy of the factory-installed system. With the aid of some BIOS-y software, the machine can be reverted back to its factory state. Yes, all user data, all patches that have been applied, EVERYTHING will go. And, of course, it doesn't help if the hard disk is what died.

You may also have "high-end" vendor backups. The "Rescue and Recovery" system that comes with Thinkpads may be an example. These tools tend to be aimed at getting the machine working again (at the expense of your data?); your mileage may vary.

Medium

For file backups, scribbling into the 'net (e.g. to Mozy or to your Unix home directory back at base) is an option. Note: pushing and pulling gigabytes to/from a faraway network place will be slow and painful.

USB memory sticks may also be adequate (if you don't have too much to back up).

CDs and DVDs are too much trouble -- except for producing that all-important you-can-boot-from-it disk. Make a couple, stick 'em in a drawer (or two), and then go get an external drive.

The workhorse of backups these days should be that external disk drive. Cheap enough, convenient and high capacity.

Don't get a "NAS" (Network Attached Storage) doodah -- it's just a crippled Linux machine you'll have to beat into submission.

You want a disk and an enclosure with a USB cable (unless you are worried about USB bandwidth).

For an enclosure that is going to be on all the time (how else are you going to have unattended backups?), I would suggest one with a power brick (not USB-powered) and with a fan. An example (probably out-of-date already): a Tsunami ST3000. Works with any old 3.5-inch SATA drive.

Image backups: some choices

(1) The market leader for this stuff is Acronis True Image Home.

In theory, it can do everything I want (both image and file backups, both full and differential, etc.). My "trial" was not entirely successful:

• I tried to set up two scheduled thingies (daily), one to do an image (differential) backup, and another to an incremental file backup. I was never sure that both happened.

• As far as I can tell, the image backup thing mostly crashed; you know, the "Do you want to send a report to Microsoft...?" thing. Not confidence-inspiring.

• I wanted both jobs to send me email about their action. Only one of them ever did.

• I never found out what happens when the disk-holding-backups fills up. I looked hard at the manual (missed it?), and I asked their support web site (they never answered).

That said, there are many satisfied users. Maybe it was Just Me.

(2) The other market leader is probably Symantec Ghost. I haven't tried it.

(3) There are at least two "we're cheaper than Acronis" options, Paragon Drive Backup and DriveImage XML. I haven't tried either.

The DriveImage XML free version looks particularly useless because they don't give you the (no doubt...) Linux live CD with which to do a bare-metal restore.

(4) In the freeware world, we have Clonezilla. It's really about imaging many machines, but backup and restore is a special case. And I think you have to take the machine down to take the image. The non-native English in the docs didn't help me...

File backups: some choices

Any full-on backups program will do file backups; for instance, the Acronis package I tried did so (as well as doing image backups).

There are zillions of file-backups programs. (Maybe you have one you particularly like.)

One system offering online file backups is Mozy Pro. (This deals with the "lost granny's letter while in Korea" problem.)

A fairly-typical (and apparently decent?) file-backups program is SyncBack, in its various incarnations.

The low-tech freeware two-way synchronizing software is Unison. I use it all the time, but it seems to be an acquired taste.

What you can do now

1. Do something. Don't put it off.

2. If the machine vendor provided a way to create a boot or recovery disk (standard with Vista SP1, apparently), do it now.

   (OK, that's nothing much to do with backups, but it's a start.)

3. Take at least one "image backup".

4. For "one image backup", CD/DVDs will work as media.

   For a lower-hassle long-term solution, back up to an external hard drive.

5. Make at least one bootable CD which you could use to restore your image backup to a new hard drive.

6. Test what you've got, at least a little, e.g. boot once with your "one bootable CD".

7. If your tool offers a confidence-building option, e.g. sending you email once a day, use it.

[An earlier version of this note appeared in Verilab's internal newsletter.]

More than once, when attending to an ill PC, I have needed to boot from a CD (e.g. with a rescue disk) but the machine showed no willingness to do so. This is very bad, because boot-from-CD is now the assumed save-the-day mechanism.

I am talking about the case where the CD drive works (based on other evidence) and where you've checked all the BIOS-y boot-order stuff, and still no joy.

This problem is more likely to arise on older hardware, which means (a) you may get no further forward booting from a USB stick, and (b) the machine may have a working floppy drive. (And any sane sysadmin still keeps a USB floppy drive in the bottom desk drawer, Just In Case [TM], no...?)

Two things.

First, no-boot-from-CD may be an intermittent problem. Try it a few times, sometimes from hot (machine already on) and sometimes from cold (machine powered off). You may be pleasantly surprised.

Second, you might want to try the Smart BootManager. Use it to produce a boot-manager-and-nothing-else on a floppy. Boot from that, and it will show you your boot-from-it options.

At first, I thought it was the combination of the SBM floppy and a bootable CD that made it work. But I think it was really the 'intermittent' thing...

If boot-from-CD really isn't going to happen, and you can't change the drive (which may not help anyway), then you are in the realm of Things You Can Do With Floppies. No fun at all.