September 2009 Archives

Complex Subroutines in GNU Make

| No Comments

Another in a very short "series" about make. If you need a small make refresher, see this earlier entry.

Here we consider a way to write complex (multi-parameter) "subroutines" for GNU Make; this is important for code reuse.

Introduction

foo.o : foo.c
    gcc -O -c foo.c -o foo.o

"If foo.c is newer than foo.o, rebuild foo.o by running gcc -O -c foo.c". Files. Timestamps. Scripts (Bourne shell).

If make were nothing but that, it would be useful but turgidly repetitive. For example, in the Makefile for a C program with 100 C files, you would have 100 nearly-identical stanzas like the example above.

If you decided to change from gcc -O to gcc -O2, you would have 100 stanzas to change.

The first way we improve things is to lift out the constants, using make variables; our example stanza might then become:

CC = gcc
CFLAGS = -O

foo.o : foo.c
    $(CC) $(CFLAGS) -c foo.c -o foo.o

Now we can change from -O to -O2 with a single keystroke.

Make also has a bunch of "automatic variables", which take on certain values in the context of a specific target. The most important are:

$@  the name of the target
$<  the name of the first prerequisite

Our example now becomes:

foo.o : foo.c
    $(CC) $(CFLAGS) -c $< -o $@

Simple Subroutines

Imagine we were concerned with compiling two C files; we might have:

foo.o : foo.c
    $(CC) $(CFLAGS) -c $< -o $@
bar.o : bar.c
    $(CC) $(CFLAGS) -c $< -o $@

The two stanzas are identical except for the target-file "stem". Make lets us common-up all such stanzas with a single pattern rule:

%.o : %.c
    $(CC) $(CFLAGS) -c $< -o $@

The bit of the filename matched by the % in the pattern is called the "stem" and is available in the automatic variable $*; so the rule could also be written as:

%.o : %.c
    $(CC) $(CFLAGS) -c $*.c -o $*.o

A useful way to think of a pattern rule is as a "make subroutine" with exactly one parameter (the stem), which you have to hide in the target filename. (Pretty horrible programming-language syntax, but...)

Pattern rules are a good idea in 100% of the cases where they apply.

Complex Subroutines

In our EDA world, tool invocation is rarely simple enough to be susceptible to the wiles of a ("single parameter") pattern rule.

Here's how you might compile a bunch of Verilog with Icarus:

a.out : $(SRCS_V)
    iverilog -o $@ -Idir1 -Idir2 -y dir1 -y dir2 $(SRCS_V)

After we lift out constants, we might get something like (and this is still simplified compared to Real Life...):

IVL     = iverilog
INCDIRS = dir1 dir2
LIBDIRS = $(INCDIRS)

a.out : $(SRCS_V)
    $(IVL) -o $@ $(INCDIRS:%=-I%) $(LIBDIRS:%=-y%) $(SRCS_V)

(Note: $(INCDIRS:%=-I%) means "take the value of the variable INCDIRS and put a -I in front of every word".)

The problem here is that the "rule" for compiling Verilog with Icarus has several "parameters":

  • the name of the desired output (a.out)
  • the include directories required (INCDIRS)
  • the library directories required (LIBDIRS)
  • the source files to feed in (SRCS_V)

There could easily be a few more. There is simply no way to squeeze all those parameters through a normal make pattern rule at once. (Well, you could... You could encode all needed information in the target name. You might say:

compile : compile++a.out++INCDIRS=dir1+dir2++LIBDIRS=dir1+dir2++

and then have a pattern rule:

compile% : # now $* includes all the extra info
    <fancy script that unpicks $* and then does what it likes>

But that's really, really horrible!)

What we want is to be able to vary the value of the "parameter variables" depending on the target that make is trying to rebuild. Happily, GNU Make provides an obscure feature to do this: target-specific make variables.

I propose the convention that target-specific variables are in lower case, so they stand out from regular variables which are (by convention) in upper case.

You can make a very simple Makefile to see target-specific variables in action:

a : foo = a_value
b : foo = b_value

a :
    @echo foo=$(foo)
b :
    @echo foo=$(foo)
c :
    @echo foo=$(foo)

When you run make a b c, you will see:

foo=a_value
foo=b_value
foo=

Back to our Icarus Verilog example. We might end up with a rule something like:

%.icarus : $(srcs_v)
    $(IVL) -o $@ $(incdirs:%=-I%) $(libdirs:%=-y%) $(srcs_v)

This is, in effect, a "subroutine" for GNU Make with three parameters. We can now invoke this "subroutine" for two (or more) different Icarus runs:

foo.icarus : srcs_v  = foo_top.v
foo.icarus : incdirs = include
foo.icarus : libdirs = lib

bar.icarus : srcs_v  = bar_top.v bar_extra.v
bar.icarus : incdirs = wibble wobble
bar.icarus : libdirs = $(incdirs)

Nobody said it was pretty, but it does mean that an arbitrarily-complex script for doing a piece of an EDA flow can be captured by a pattern rule (using target-specific make variables) -- in exactly one place. This can only be good for increasing the robustness of any make-based build system.

[An earlier version of this note appeared in Verilab's internal newsletter.]

The Humble Make Stanza

| No Comments

I once had a spell looking at a lot of Makefiles. It prompted this ode to the Humble Make Stanza.

A make Refresher

Make is a tool to control the rebuilding of files so that unnecessary work is avoided. The simplest construct in make is a stanza:

target(s) : prerequisite(s)
    script(s)

Meaning: If the target file doesn't exist or if one of the prerequisite files is newer, then rebuild the target file(s) by running script(s). Here's a specific example:

foo.o : foo.c
    gcc -O -c foo.c -o foo.o

"If foo.c is newer than foo.o, rebuild foo.o by running gcc -O -c foo.c".

There's really no magic. You could just as easily write:

foo.o : foo.c
    echo my granny wears army boots

which is barking mad, but absolutely fine by make.

Files. Timestamps. Shell scripts (Bourne shell). What could be simpler?

make in Full Glory

The ideal make stanza does a single step in a potentially complex workflow. make stitches together many such stanzas into a beautiful harmonious whole. Let us remind ourselves of what we are hoping for:

  • The make infrastructure will capture the whole workflow. Type make world and the full workflow bhoona will unfold before your eyes, the right steps in the right order.

  • When something changes, the steps that must be re-done will definitely be re-run. (Anything else is a disaster.)

  • When something changes, only the steps that must be re-done will be re-run. Not essential but preferred. For example: it's a pain that, when you change a comment in a header file, make will re-compile great swathes of source code. We live with it.

  • When something goes wrong, make will not proceed pointlessly (e.g. do a doomed seven-hour simulation even though the preceding compilation failed) and will make it relatively clear where matters came unstuck.

  • If a previous run was interrupted, make will restart at the right place (and not be confused).

  • If the IT infrastructure has provision for parallel simulations (e.g. a batch queuing system), make will use that appropriately.

The Naked Stanza

You will undoubtedly not recognize your own experience with make from the preceding idyllic description. To understand why, let's look more closely at the soul of the make party, the stanza:

target(s) [hidden t's] : prerequisite(s) [hidden prereq's]
    script(s)

The core task to be performed by the script -- e.g. "run VCS with these command-line arguments" -- is often straightforward and easy to get right. So where do problems arise?

  • The script(s) will read a bunch of files -- prerequisite(s) -- and probably write at least one -- target(s).

    Unless carefully done, the scripts will probably have hidden prerequisites (files read but not specifically listed in the Makefile) and hidden targets (files written but similarly unlisted).

  • If you don't attend to the "hidden" prerequisites (e.g. by buying ClearCase and then using clearmake, which tracks such dependencies through "audited builds"), then you should stop using make straight away -- you have no assurance that crucial rebuilding will happen when it really needs to.

    (Of course, there is the make equivalent of the Windows re-install: in the face of the slightest hiccup, type make clean. Why someone with a lousy Makefile trusts the clean target continues to elude me, however.)

  • Hidden targets are less of an issue. They may cause needless extra rebuilding by make, but that may not be a hanging offense.

  • A problem that arises with some EDA tools is that they are disinclined to do a single unit of work (and then get out of the way). They often want to set up a hidden universe of their own (often with dubious Makefiles hidden somewhere), and/or they decide they should offer counseling before getting on with the job ("Mr Kelly, are you really sure you want to merge those two files? Click OK if you're absolutely certain you want to proceed, Cancel otherwise; this dialog box comes with no warranty expressed or implied.")

  • The script(s) needs to offer an absolute assurance to the downstream parts of the overall "flow" that they have done their part successfully. This is usually by setting the exit status correctly (zero for OK, anything else for not).

    EDA tools are imperfect on this exit-status thing. Some just don't bother to set the exit-status to anything sensible. For others (notably simulators), the tool's idea of "not OK" may reasonably differ from your own.

    For example, a simulation may run to completion (exit 0) but it may have properties which mean its results must not be passed on to the next step of the workflow.

  • The assurance of job-well-done also needs to be recorded for posterity in a file-with-timestamp (because that's make understands). This might be called a "witness" file -- it witnesses that a particular task was done successfully at a certain time.

    In many cases such as the .c-to-.o example given at the beginning, the .o file is the witness (as well as the object file that you really want). In its witness capacity, it is saying "A successful C compilation of foo.c took place on November 22 at 11:12am."

    In the more complex EDA world, I tend to favor explicit witness (or "stamp" or "sentinel") files: sim-4732-DONE witnesses that simulation 4732 completed successfully at the time indicated.

  • Another thing the scripts need to do is leave a trail of what happened. This is often in the form of log files.

  • Even better than "what happened" is "what happened that was interesting". A definite weakness of EDA flows is that they spew voluminously, and it is very easy to miss the One Crucial Message that shows something amiss.

The Model Stanza

The observations above hint at what a "normal" make stanza should look like in an EDA -- or similarly complicated -- flow. Here it is:

witness-file : prerequisite(s)
    /bin/rm -f witness-file log-file
    eda-tool ...args... 2>&1 | tee-grep log-file
    update-dependencies
    decide-status log-file
    /bin/touch witness-file

-include witness-file.P  # dependencies from update-dependencies
  • The goal is to produce the witness-file; if it exists, it means "this steps completed successfully at the time indicated".

  • eda-tool ...args... is just the normal command-line invocation of the EDA tool.

  • We pipe the output(s) of the EDA tool to tee-grep, a (hypothetical) tool whose jobs are: (a) to squirrel away a copy of the log file for later perusal; and (b) to display only the log messages that are "interesting" (for the local definition of same).

  • If you don't have ClearCase or equivalent, you will need some special pleading to update the dependencies related to this step; I have hypothesized an update-dependencies tool, and that it puts its output into witness-file.P, which make slurps in (last line -include).

  • All of the scripts before decide-status should have completed successfully (exit status zero). (If not, something bad is wrong, e.g. out of disk space.) The question remains: can workflow steps that are relying on this one "succeeding" go ahead? That's what the decide-status script does: exit status zero will mean "go ahead" (after stamping the witness-file), non-zero exit means "go no further".

Make has a reputation for being flaky and hard to work with. If each make stanza were as bullet-proof as suggested here, most of that problem would go away.

[An earlier version of this note appeared in Verilab's internal newsletter.]

Hands off my boot boot

| No Comments

I did a Fedora 11 install and, not unusually, the machine wouldn't boot afterward -- GRUB bootloader gunk messed up.

Attempted a normal remedy: Boot "rescuely" with the install CD, do chroot /mnt/sysimage and run...

grub-install --no-floppy --root-directory=/boot '(hd0)'

(This machine is all software RAID-1, with a separate /boot partition.)

I reboot... into a GRUB prompt (sigh). It has not found the grub.conf (or menu.lst -- I never know which one it really looks for). It is suspiciously easy to get it to do the right thing:

grub> configfile (hd0,0)/grub/grub.conf

Hmm... How is that different from what GRUB should have done on its own?

Turns out that grub-install didn't do the right -- or expected -- thing. It dropped its stuff inside /boot, so I ended up with /boot/boot/grub -- but with no grub.conf menu file in that directory. Working GRUB but no menu -- the symptoms seen.

A GRUB install done the low-tech (interactive) way sorted things:

grub> root (hd0,0)
grub> setup (hd0)

About this Archive

This page is an archive of entries from September 2009 listed from newest to oldest.

August 2009 is the previous archive.

October 2009 is the next archive.

Find recent content on the main index or look in the archives to find all content.