We are a small company with people hither and yon (for various reasons) who need to see/work-on the same "documents". In a normal Windows Explorer kinda way (not some over-there-in-the-browser kinda way) and with (near-)LOCAL DISK speed. (I do not want to hear the word 'WebDAV' ever again.)

Yes, they always need to see the "latest" version at all times. (No, they do not want to click an 'Update' button.)

They need to be told "Whoa! Two of you are trying to write that thing at once!"

Different (changing) groups of people need to see different groups of documents. Only finance people can see finance documents, etc.

An "admin" needs to be able to login, get some oversight of things, and change stuff, e.g. shut off the person who just left the company.

Windows, Macs and Linux, please. (OK, OK... we're flexible.)

I would have thought the above would be the FIRST THING any "cloud storage" wannabee might turn his hand to.

Obvious "solutions": NONE (that I've found). More as I find it.

Now, I don't like April 1 jokes that are apparent just from the title; but I erred too far on the side of obscurity.

A few weeks ago, we had email chat at work about some of the weirdness you see on your router if Skype is running. I mused that I would prefer that people who do not use Skype should turn it off. Each time I was reminded that Skype was a major communication tool within Verilab.

So last Thursday I "announced" that a Skype ban would be rolled out starting with people "from the middle of the alphabet". People who don't even work in that office were "banned". To summarize:

• The "ban" was high-handed and made entirely without consultation.

• No alternative was proposed, making the "ban" extremely business-hostile.

• The "analysis" on which the "ban" was based -- we occasionally see bursts of UDP packets for a minute or two -- was, errmm... "limited" to say the least.

• "Starting in the middle of the alphabet" -- say what?

• Summary: COMPLETELY INSANE.

What's interesting is that a COMPLETELY INSANE proposal from the IT guy is taken as quite normal. I was asked polite questions, and people seemed to be trying to adapt. I don't know whether my colleagues intended to ignore me, work around me, or what. (Of course, it may be a double-bluff and the joke's on me.)

Perhaps we see so much COMPLETELY INSANE stuff in all the organizations we bump up against in everyday life that it seems normal for the IT guy to trash the company's communication structure.

Microsoft's apparent attitude to this sullen army of volunteer helpers: No good deed goes unpunished.

Case in point. Some friends' PC died; they got a new one; the disk out of the old one was fine and readily sprang to life in a USB external enclosure. They'd like to move their old stuff over to the new machine.

That would be a few minutes' easy work under Linux (but this is Windows).

Ah! But what's this?! "Windows Easy Transfer" [WET] (under Windows 7) -- "Helps you transfer personal files, e-mail, data, files, media, and settings from your old computer to the new one." Fantastic! And it can read from an external USB drive -- glory be!

Sadly, you have to have "prepared" for the "Easy Transfer" by running WET on the old machine first -- presumably before the puff of smoke and the acrid smell in the air.

SIGH. And does WET have any sort of second-best fallback option? e.g. "We can't transfer your user accounts, but we can at least move some documents over for you?" (You know, the equivalent of a one-line shell script.)

No, of course not. (What, exactly, do all of those "programmers" in Redmond do all day every day?)

So I'm doing Windows Tedious Transfer [WTT]. Copying files/folders around, deleting Obviously Useless things (e.g. cookie files), and so on. I.e. doing a third-rate job taking eight times as long.

No good deed goes unpunished.

And what should you do on the guest (the VM itself)?

Well, following the cited scheme, the VM will be handed an LVM "logical volume" to use.

The VM will treat those bits as a disk (as it would if you created a big file and handed that over, instead). Your only real choice is to partition that virtual disk, and use the resulting partitions in the usual ways.

If you know exactly how to use the VM's disk space, and "it will never change", then create fixed partition(s), slap filesystem(s) on them, and you're done.

If, however, we again assume a need for "flexibility", then LVM is again a good idea -- yes, LVM(guest)-within-LVM(host).

I have been told (but not tried myself) that LVM on the host can "see" the inner-LVM stuff (the guest's) and do things with it, e.g. snapshot it.

(Apparently important: the volume group(s) on the guest must have different names from the volume group(s) on the host.)

But the main virtue of LVM-within-LVM is that it's relatively simple to increase the "disk space" on the guest...

Stop the VM: virsh shutdown guest

On the host, extend the logical volume that is the guest's "disk":

lvextend -L20G /dev/vg0/guest-disk

Restart the VM: virsh start guest

The guest's "disk" just got magically bigger. Use fdisk (or equiv) to make a new "partition" out of the extra space, giving it the LVM type (8e).

Make the new "partition" an LVM "physical volume" (PV):

pvcreate /dev/xvda3

Add the PV to the VM's "volume group" (VG):

vgextend guestvg0 /dev/xvda3

Now you can make one of the "inner" (guest) "logical volumes" (LVs) bigger:

lvextend /dev/guestvg0/lvroot ...

Finally, the filesystem sitting on that LV will need resizing:

resize2fs /dev/guestvg0/lvroot

First: disks are cheap enough that I always use two, and software-RAID-1 them together (mirroring). Plan to do that.

Second: Though I'm no fan of LVM (the Logical Volume Manager) -- extra complication for no gain -- it's a good choice for "host for a bunch of VMs (virtual machines)". I'm assuming you don't know exactly which VMs of what size, so the flexibility of LVM is actually useful.

Also, as disks get larger, it's harder to predict in advance how you'll end up using all that space. So, again, the flexibility of LVM may benefit.

Now, I don't use LVM for the basic filesystems (/boot and / [root]). Their sizes are predictable and, when It Goes Horribly Wrong (TM), you'll be really glad not to have LVM in the way. So I partition my raw disks thusly:

Filesystem  Size  Used for
/dev/md0    300M  /boot
/dev/md1     16G  /
swap         ???
/dev/md2    rest  of the disk(s)

That /dev/md2 goes straight into LVM. Make a PV (Physical Volume):

pvcreate /dev/md2

And create a Volume Group (let's call it vg0) with it:

vgcreate -s 32M vg0 /dev/md2

(That -s 32M is not particularly well-informed.)

Now you can make LVs (Logical Volumes) to your heart's content...

lvcreate -n vm001-disk --size 10G vg0

... and feed them to your VM-creating software. (Or make things for your host machine, e.g. a varying-sized /var.)

A word on LVM-on-RAID... LVM has a mirroring facility of its own, so why use software RAID1?

First, I'm already paying the software-RAID tax with my non-LVM partitions (/boot and /).

Second: For me, software RAID is familiar and has been reliable.

Third: Software RAID-1 is less scary in a crisis. I like things like: You can take a disk that is half of a RAID-1 mirror, throw it in an enclosure, and mount it as a filesystem, no problem.

Fourth: The LVM mirroring guff seemed more complicated and/or easier to get wrong. (Just unfamiliarity, maybe.)

Running ionice -c3 /bin/rm ... should mean that rm "will only get disk time when no other program has asked for disk I/O..." (man page). That sounds about right.

Of course, ionice will not stop seek storms, nor will it leave the disk heads in a place convenient for other programs that are trying to do Real Work (TM).

NB: I have no empirical evidence that ionice actually helps in cases where /bin/rm renders a machine otherwise useless.

I made the disastrous choice to remove some of those directories, you know... /bin/rm -rf *.OLD.

Two weeks later, the job isn't finished.

The job probably would be finished if I had exclusive use of the machine. But the other users get stroppy when I render the machine useless to them with my exotic Power Tool, /bin/rm. The load average runs to 5, the CPU romps up to 98+% IO waiting, and the other users start calling my number. Not in a nice way.

I would have thought that removing files-and-folders was about the most settled/debugged/worked-out part of Linux. How naive I obviously am.

My situation, or one very like it, was discussed on the Linux kernel list: "Very poor ext3 write performance on big filesystems?" (Probably elsewhere, too.) You can read it for yourself.

The first thing I did was make a script rm-and-sleep; not rocket science:

#!/bin/sh
/bin/rm ${1+"$@"}
sleep 5

I can then run that with something like...

find . -type f -print | xargs -n 50 rm-and-sleep

... and be sure I am not eating all of the system all of the time. (Just seeing the script makes me want to weep... decades of computing progress, and we're doing this?)

One thing suggested in the discussion cited above is to remove things in inode order. I ended up using cut rather than awk in Chris Mason's script, giving:

find . \( \! -type d \) -printf "%i %p\n" \
| sort -n \
| cut -d ' ' -f 1 --complement \
| xargs -n 500 rm-and-sleep

Guff to deal with spaces-in-filenames and nice and so forth -- omitted for brevity.

Ugh. Ugh. Ugh.

debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY

The only hint in the apparently-most-relevant online discussion is that it might have something to do with some network interface's MTU (Maximum Transmission Unit).

Lacking any better ideas, we changed the MTU on the SSH server (with ifconfig eth0 mtu 512) -- note: a value wildly lower than what should be required -- and it worked! SSH connections could happen.

Except: only just barely. The first time the remote end tried to send any amount of data -- e.g. a VNC screen refresh -- the connection would wedge, and that would be that. (I cannot explain this data point: I would expect the very low MTU setting to ensure that no big packets ever got out, and so no wedging could happen on that account.)

On the weak assumption that our problem was something to do with MTUs, I proceeded to learn a fair bit more than I wanted to know.

Every network link has an MTU, and there are on-the-order-of a dozen links between Here and There in any network interaction. An MTU of 1500 is standard for Ethernet (modulo Jumbo Frames), but you get MTUs in the high 1400s for various network technologies (1500 minus "some bytes for our special stuff"). Needing an MTU under 1000 means "something is not right".

What happens if you send a packet that is too big to make the next network hop? In theory, the network device is allowed to "fragment" the packet, and send on the pieces. I believe this sort of malarkey is frowned on, and many (most? (all?)) TCP packets are sent with a DF (Do Not Fragment) flag.

So what happens with a DF-flagged too-big packet? Well, the device at the edge of the too-small-MTU chasm is supposed to send back an ICMP packet telling of the problem, so that the originator can re-send in smaller bundles.

A whole bunch of things can go wrong. The device might simply fail to send the required ICMP packet. Any of the network devices on the way back to the originator might choose to drop the ICMP packet ("Security, you know."). The host firewall -- or the one on the last-hop consumer router -- stands a high chance of Doing The Wrong Thing.

Net effect: Big packets go, but then just drop off the end of the world. Re-sending doesn't help because, odds are, they will follow the same route and drop no less ignominiously. Eventually the connection will time out.

Your packets are said to have fallen into an MTU black hole.

Along the way, I learned that you can probe network MTUs by sending various-sized ping packets. (As in most networking problems ping and traceroute are your friends.) In Linux-speak, you type...

ping -M do -c 2 -s <n> remote.example.com

...varying <n>. So, for instance, if

ping -M do -c 2 -s 1465 remote.example.com

balks with Frag needed and DF set, but

ping -M do -c 2 -s 1464 remote.example.com

does not, then 1464+28=1492 is the least MTU between here and there. (ping has 28 bytes overhead which you need to add back in.)

Back to our own sad case. Besides trying our fabulously-low MTU of 512, we did all the other typical things, namely power-cycling all the networking doodahs around the office. In desperation, we even did a Linux reboot (a mistake). Nothing helped.

But when we woke up next morning and tried our stuff, everything was fine. Completely baffling.

Let's say it's 192.168.0.1 in the faraway office. So: SSH in with tunnel...

ssh -L10080:192.168.0.1:80 me@faraway.example.com

Now use your own web browser to visit http://localhost:10080/; it should be your faraway router's web interface.

Just another use of Fabulous SSH Tunneling...

(A similar tool I use quite a bit is hardlink, which doesn't tell you about duplicate files; it just hard links them together. Which can be good.)

What I've really wanted, though, is a tool to which I could say "See that pile #1 of directories over there? And that pile #2 over there? Tell me what files are in pile #1 that aren't in pile #2." Or vice versa. Or maybe "the files that are in both piles".

What I don't want to know is that there are four copies of a file in piles #1 and #2 -- because they might all four be in pile #1.

For the special case of "files arranged identically in various directories" (e.g. if those directories are backup copies), here's something that comes close.

Over pile #1 of directories, run...

find <dirs> -type f -print | xargs sha1sum > my-file-list

The output, my-file-list, might need massaging.

Now go over to pile #2 of directories and run...

sha1sum -c my-file-list

The goal is: like-named identical files should be flagged up as 'OK', and the others as something else.

Let's say you want to delete those like-named identical files. It would then be something like (WARNING: UNTESTED)...

sha1sum -c my-file-list | egrep ': OK$' \
| sed -e 's/: OK$//'    | xargs -r /bin/rm

(I always grow such scripts incrementally, and by hand. You'll have to anyway if you've got filenames-with-spaces, or other fun.)

We decided to start with a magnificent "grid" of... cough ONE NODE. yum install the packages, ./install_qmaster, ./install_execd and -- bingo! -- major "grid" action!

Would that it were so simple. Any question in plain English -- "How do I get it to run four jobs at once?" or "How do I change the load threshold at which new jobs start to 1.1?" -- will be met by an avalanche of Grid Engine Speak ("... shows values for those attributes that are defined as per queue instance slot limits or as fixed resource attributes").

Our big mistake, however, was to expand our "grid" to... cough TWO NODES.

Much the same drill -- yum install, ./install_execd -- and it seemed to work, except when it doesn't. The most baffling message -- on the original node that worked perfectly by itself -- is:

failed on host foo.verilab.com general before job because:
10/16/2009 19:05:35 [0:31072]: setuid(547) failed

Don't even know where to look. Finally decided to ask on the "gridengine-users" mailing list.

In Ye Olden Dayes, you'd just send email to the list, and maybe be held in the moderator queue the first time.

Now, I've registered for the-deities-know-what with Sun, and signed up as a project "observer", all so I can ask my question. Which was still held for "Pending Approval", as was my follow-up.

So, for a week's effort, I've been asked "Are you running it as root?" (Er... I thought the point of setuid was for lower-privileged users to do higher-privileged things... but obviously I haven't a clue.)

Don't even know where to look.

Potential Grid Engine users: you may find make -j 4 is easier.

The shape of this script is roughly...

svnlook ... | while ; do ...check... ; done

The svnlook-ish bit produces filenames to look at, and the "check" inside the while runs puppet --parseonly, which grumbles and gives a non-zero exit code if there's a syntax error.

The script-as-a-whole needs to give a non-zero exit status if there is a problem, i.e. "don't commit".

The script is burst because it's hard to get information from inside the while loop so that a script-as-a-whole decision can be made.

Why? Because the parts of a Unix pipeline are run as sub-shells, meaning the while loop is all in a sub-shell. Any variable-setting and/or exiting in the loop will only affect the sub-shell and make absolutely no difference to the script-as-a-whole.

Moral: while in a pipeline -- probably a bad idea.

(Just ask if you want my pre-commit script. Mine checks .erb template files, too.)

Here we consider a way to write complex (multi-parameter) "subroutines" for GNU Make; this is important for code reuse.

Introduction

foo.o : foo.c
    gcc -O -c foo.c -o foo.o

"If foo.c is newer than foo.o, rebuild foo.o by running gcc -O -c foo.c". Files. Timestamps. Scripts (Bourne shell).

If make were nothing but that, it would be useful but turgidly repetitive. For example, in the Makefile for a C program with 100 C files, you would have 100 nearly-identical stanzas like the example above.

If you decided to change from gcc -O to gcc -O2, you would have 100 stanzas to change.

The first way we improve things is to lift out the constants, using make variables; our example stanza might then become:

CC = gcc
CFLAGS = -O

foo.o : foo.c
    $(CC) $(CFLAGS) -c foo.c -o foo.o

Now we can change from -O to -O2 with a single keystroke.

Make also has a bunch of "automatic variables", which take on certain values in the context of a specific target. The most important are:

$@  the name of the target
$<  the name of the first prerequisite

Our example now becomes:

foo.o : foo.c
    $(CC) $(CFLAGS) -c $< -o $@

Simple Subroutines

Imagine we were concerned with compiling two C files; we might have:

foo.o : foo.c
    $(CC) $(CFLAGS) -c $< -o $@
bar.o : bar.c
    $(CC) $(CFLAGS) -c $< -o $@

The two stanzas are identical except for the target-file "stem". Make lets us common-up all such stanzas with a single pattern rule:

%.o : %.c
    $(CC) $(CFLAGS) -c $< -o $@

The bit of the filename matched by the % in the pattern is called the "stem" and is available in the automatic variable $*; so the rule could also be written as:

%.o : %.c
    $(CC) $(CFLAGS) -c $*.c -o $*.o

A useful way to think of a pattern rule is as a "make subroutine" with exactly one parameter (the stem), which you have to hide in the target filename. (Pretty horrible programming-language syntax, but...)

Pattern rules are a good idea in 100% of the cases where they apply.

Complex Subroutines

In our EDA world, tool invocation is rarely simple enough to be susceptible to the wiles of a ("single parameter") pattern rule.

Here's how you might compile a bunch of Verilog with Icarus:

a.out : $(SRCS_V)
    iverilog -o $@ -Idir1 -Idir2 -y dir1 -y dir2 $(SRCS_V)

After we lift out constants, we might get something like (and this is still simplified compared to Real Life...):

IVL     = iverilog
INCDIRS = dir1 dir2
LIBDIRS = $(INCDIRS)

a.out : $(SRCS_V)
    $(IVL) -o $@ $(INCDIRS:%=-I%) $(LIBDIRS:%=-y%) $(SRCS_V)

(Note: $(INCDIRS:%=-I%) means "take the value of the variable INCDIRS and put a -I in front of every word".)

The problem here is that the "rule" for compiling Verilog with Icarus has several "parameters":

• the name of the desired output (a.out)
• the include directories required (INCDIRS)
• the library directories required (LIBDIRS)
• the source files to feed in (SRCS_V)

There could easily be a few more. There is simply no way to squeeze all those parameters through a normal make pattern rule at once. (Well, you could... You could encode all needed information in the target name. You might say:

compile : compile++a.out++INCDIRS=dir1+dir2++LIBDIRS=dir1+dir2++

and then have a pattern rule:

compile% : # now $* includes all the extra info
    <fancy script that unpicks $* and then does what it likes>

But that's really, really horrible!)

What we want is to be able to vary the value of the "parameter variables" depending on the target that make is trying to rebuild. Happily, GNU Make provides an obscure feature to do this: target-specific make variables.

I propose the convention that target-specific variables are in lower case, so they stand out from regular variables which are (by convention) in upper case.

You can make a very simple Makefile to see target-specific variables in action:

a : foo = a_value
b : foo = b_value

a :
    @echo foo=$(foo)
b :
    @echo foo=$(foo)
c :
    @echo foo=$(foo)

When you run make a b c, you will see:

foo=a_value
foo=b_value
foo=

Back to our Icarus Verilog example. We might end up with a rule something like:

%.icarus : $(srcs_v)
    $(IVL) -o $@ $(incdirs:%=-I%) $(libdirs:%=-y%) $(srcs_v)

This is, in effect, a "subroutine" for GNU Make with three parameters. We can now invoke this "subroutine" for two (or more) different Icarus runs:

foo.icarus : srcs_v  = foo_top.v
foo.icarus : incdirs = include
foo.icarus : libdirs = lib

bar.icarus : srcs_v  = bar_top.v bar_extra.v
bar.icarus : incdirs = wibble wobble
bar.icarus : libdirs = $(incdirs)

Nobody said it was pretty, but it does mean that an arbitrarily-complex script for doing a piece of an EDA flow can be captured by a pattern rule (using target-specific make variables) -- in exactly one place. This can only be good for increasing the robustness of any make-based build system.

[An earlier version of this note appeared in Verilab's internal newsletter.]

A make Refresher

Make is a tool to control the rebuilding of files so that unnecessary work is avoided. The simplest construct in make is a stanza:

target(s) : prerequisite(s)
    script(s)

Meaning: If the target file doesn't exist or if one of the prerequisite files is newer, then rebuild the target file(s) by running script(s). Here's a specific example:

foo.o : foo.c
    gcc -O -c foo.c -o foo.o

"If foo.c is newer than foo.o, rebuild foo.o by running gcc -O -c foo.c".

There's really no magic. You could just as easily write:

foo.o : foo.c
    echo my granny wears army boots

which is barking mad, but absolutely fine by make.

Files. Timestamps. Shell scripts (Bourne shell). What could be simpler?

make in Full Glory

The ideal make stanza does a single step in a potentially complex workflow. make stitches together many such stanzas into a beautiful harmonious whole. Let us remind ourselves of what we are hoping for:

• The make infrastructure will capture the whole workflow. Type make world and the full workflow bhoona will unfold before your eyes, the right steps in the right order.

• When something changes, the steps that must be re-done will definitely be re-run. (Anything else is a disaster.)

• When something changes, only the steps that must be re-done will be re-run. Not essential but preferred. For example: it's a pain that, when you change a comment in a header file, make will re-compile great swathes of source code. We live with it.

• When something goes wrong, make will not proceed pointlessly (e.g. do a doomed seven-hour simulation even though the preceding compilation failed) and will make it relatively clear where matters came unstuck.

• If a previous run was interrupted, make will restart at the right place (and not be confused).

• If the IT infrastructure has provision for parallel simulations (e.g. a batch queuing system), make will use that appropriately.

The Naked Stanza

You will undoubtedly not recognize your own experience with make from the preceding idyllic description. To understand why, let's look more closely at the soul of the make party, the stanza:

target(s) [hidden t's] : prerequisite(s) [hidden prereq's]
    script(s)

The core task to be performed by the script -- e.g. "run VCS with these command-line arguments" -- is often straightforward and easy to get right. So where do problems arise?

• The script(s) will read a bunch of files -- prerequisite(s) -- and probably write at least one -- target(s).

  Unless carefully done, the scripts will probably have hidden prerequisites (files read but not specifically listed in the Makefile) and hidden targets (files written but similarly unlisted).

• If you don't attend to the "hidden" prerequisites (e.g. by buying ClearCase and then using clearmake, which tracks such dependencies through "audited builds"), then you should stop using make straight away -- you have no assurance that crucial rebuilding will happen when it really needs to.

  (Of course, there is the make equivalent of the Windows re-install: in the face of the slightest hiccup, type make clean. Why someone with a lousy Makefile trusts the clean target continues to elude me, however.)

• Hidden targets are less of an issue. They may cause needless extra rebuilding by make, but that may not be a hanging offense.

• A problem that arises with some EDA tools is that they are disinclined to do a single unit of work (and then get out of the way). They often want to set up a hidden universe of their own (often with dubious Makefiles hidden somewhere), and/or they decide they should offer counseling before getting on with the job ("Mr Kelly, are you really sure you want to merge those two files? Click OK if you're absolutely certain you want to proceed, Cancel otherwise; this dialog box comes with no warranty expressed or implied.")

• The script(s) needs to offer an absolute assurance to the downstream parts of the overall "flow" that they have done their part successfully. This is usually by setting the exit status correctly (zero for OK, anything else for not).

  EDA tools are imperfect on this exit-status thing. Some just don't bother to set the exit-status to anything sensible. For others (notably simulators), the tool's idea of "not OK" may reasonably differ from your own.

  For example, a simulation may run to completion (exit 0) but it may have properties which mean its results must not be passed on to the next step of the workflow.

• The assurance of job-well-done also needs to be recorded for posterity in a file-with-timestamp (because that's make understands). This might be called a "witness" file -- it witnesses that a particular task was done successfully at a certain time.

  In many cases such as the .c-to-.o example given at the beginning, the .o file is the witness (as well as the object file that you really want). In its witness capacity, it is saying "A successful C compilation of foo.c took place on November 22 at 11:12am."

  In the more complex EDA world, I tend to favor explicit witness (or "stamp" or "sentinel") files: sim-4732-DONE witnesses that simulation 4732 completed successfully at the time indicated.

• Another thing the scripts need to do is leave a trail of what happened. This is often in the form of log files.

• Even better than "what happened" is "what happened that was interesting". A definite weakness of EDA flows is that they spew voluminously, and it is very easy to miss the One Crucial Message that shows something amiss.

The Model Stanza

The observations above hint at what a "normal" make stanza should look like in an EDA -- or similarly complicated -- flow. Here it is:

witness-file : prerequisite(s)
    /bin/rm -f witness-file log-file
    eda-tool ...args... 2>&1 | tee-grep log-file
    update-dependencies
    decide-status log-file
    /bin/touch witness-file

-include witness-file.P  # dependencies from update-dependencies

• The goal is to produce the witness-file; if it exists, it means "this steps completed successfully at the time indicated".

• eda-tool ...args... is just the normal command-line invocation of the EDA tool.

• We pipe the output(s) of the EDA tool to tee-grep, a (hypothetical) tool whose jobs are: (a) to squirrel away a copy of the log file for later perusal; and (b) to display only the log messages that are "interesting" (for the local definition of same).

• If you don't have ClearCase or equivalent, you will need some special pleading to update the dependencies related to this step; I have hypothesized an update-dependencies tool, and that it puts its output into witness-file.P, which make slurps in (last line -include).

• All of the scripts before decide-status should have completed successfully (exit status zero). (If not, something bad is wrong, e.g. out of disk space.) The question remains: can workflow steps that are relying on this one "succeeding" go ahead? That's what the decide-status script does: exit status zero will mean "go ahead" (after stamping the witness-file), non-zero exit means "go no further".

Make has a reputation for being flaky and hard to work with. If each make stanza were as bullet-proof as suggested here, most of that problem would go away.

[An earlier version of this note appeared in Verilab's internal newsletter.]

Attempted a normal remedy: Boot "rescuely" with the install CD, do chroot /mnt/sysimage and run...

grub-install --no-floppy --root-directory=/boot '(hd0)'

(This machine is all software RAID-1, with a separate /boot partition.)

I reboot... into a GRUB prompt (sigh). It has not found the grub.conf (or menu.lst -- I never know which one it really looks for). It is suspiciously easy to get it to do the right thing:

grub> configfile (hd0,0)/grub/grub.conf

Hmm... How is that different from what GRUB should have done on its own?

Turns out that grub-install didn't do the right -- or expected -- thing. It dropped its stuff inside /boot, so I ended up with /boot/boot/grub -- but with no grub.conf menu file in that directory. Working GRUB but no menu -- the symptoms seen.

A GRUB install done the low-tech (interactive) way sorted things:

grub> root (hd0,0)
grub> setup (hd0)