GNU make amends rules with multiple targets

TL;DR

Starting from GNU make-4.4 rules with multiple targets that include commands will trigger if any of the targets does not exist. This will require a few projects to adapt. Older versions of ghc are affected.

Typical example would need to adapt from something like:

%.gen.c %.gen.h %.gen.not-always-present: %.src
	$(CMD) ...

to something like:

%.gen.c %.gen.h: %.src
	$(CMD) ...

%.gen.not-always-present: %.gen.c
	: # generated by previous rule or not generated at all

See https://savannah.gnu.org/bugs/index.php?63098 for other options.

More words

Makefile usually defines a bunch of prerequisites per single target:

foo: foo.c foo.h
	$(CC) $(CFLAGS) $(LDLIBS) foo.c -o foo

For dependency-only rules without commands it’s customary to specify multiple targets:

all: foo bar

# multiple targets
foo bar: foo.h

# equivalent to
#   foo: foo.h
#   bar: foo.h

foo:
	touch foo

bar:
	touch bar

foo.h:
	touch foo.h

In all the cases above the behaviour is straightforward: if foo.h changes then foo and bar are outdated and have to be rebuilt (if rebuild is requested). And specifically make foo should cause only foo rebuild. Example session:

$ make
touch foo.h
touch foo
touch bar

$ touch foo.h
$ make foo
touch foo

No surprise here: in a second run bar is not rebuilt and stays outdated (we did not ask for it’s update). And foo is rebuilt as expected.

In GNU make before 4.3.90 the same rule applied to rules with commands as well:

all: foo bar

foo bar: foo.h
	touch foo bar

foo.h:
	touch foo.h

Note that the command for foo bar: foo.h rule always builds both targets.

Let’s try to delete bar and ask foo to be rebuilt. Would foo get rebuilt? Would bar get rebuilt? Here is the answer:

$ make-4.3
touch foo.h
touch foo bar

$ rm bar
$ make-4.3 foo
make: 'foo' is up to date.

Looks exactly the same as above: foo does not require a refresh.

Now let’s try make-4.3.90:

$ rm -f foo bar foo.h

$ make-4.3.90
touch foo bar

$ rm bar
$ make-4.3.90 foo
touch foo bar

That’s a different behaviour: absence of bar triggers both foo and bar rebuilds. This behaviour change is intentional and is added in https://savannah.gnu.org/bugs/?62809.

The impact

So far it looks benign: we’ll build just a bit more than we used to in some incremental builds. Fresh-from-zero builds should not be affected, right? Right?

I installed fresh make-4.3.90 and attempted to build the world.

opensp case

opensp-1.5.2 being an autotools package provides tarballs with pre-generated files as part of the release:

%.h %.cxx %.rc: %.msg
	[ ! -f $(top_srcdir)/msggen.pl ] || $(PERL) -w $(top_srcdir)/msggen.pl $(MSGGENFLAGS) $<

All .h, .cxx and .rc files are already present in OpenSP-1.5.2.tar.gz. User never has to run msggen.pl script to get opensp built.

Except that msggen.pl does not always produce .cxx files. It does so only for .msg files that have a !cxx directive. I noticed it only because msggen.pl does not really work on any modern perl version (and also because nix does not expose perl to build sandbox by default). Fun fact: OpenSP-1.5.2.tar.gz was released in 2007.

The build fails on make-4.3.90 as:

$ make-4.3.90
make[2]: Entering directory '/build/OpenSP-1.5.2/lib'
[ ! -f ../msggen.pl ] || perl -w ../msggen.pl -l libModule PosixStorageMessages.msg
bash: line 1: perl: command not found
make[2]: *** [Makefile:778: PosixStorageMessages.h] Error 127 shuffle=1663959693

The proposed fix makes .cxx as optional by splitting out .cxx into a separate rule:

--- a/lib/Makefile.am
+++ b/lib/Makefile.am
@@ -125,5 +125,7 @@ SUFFIXES = .msg .m4 .rc
 .m4.cxx:
 	$(PERL) $(top_srcdir)/instmac.pl $< >$@

-%.h %.cxx %.rc: %.msg
+%.h %.rc: %.msg
	[ ! -f $(top_srcdir)/msggen.pl ] || $(PERL) -w $(top_srcdir)/msggen.pl $(MSGGENFLAGS) $<
+%.cxx: %.rc
+	: # built by perl rule above

Here we move .cxx part as a separate no-op target to avoid perl build rule from triggering. Similar fix had to be applied to a few more Makefile.am files in opensp tree.

The failure Does not look bad: it was easy to diagnose and workaround.

ghc case

ghc was another heavy GNU make user until recenty. Many distributions still package older ghc versions and still use GNU make based build system. ghc was broken by make-4.3.90 as:

$ ./configure
$ make-4.3.90
...
ghc>   HSC2HS libraries/hpc/dist-boot/build/Trace/Hpc/Reflect.hs
ghc>   HSC2HS libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap/Constants.hs
ghc>   HSC2HS libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap/InfoTable/Types.hs
ghc>   HSC2HS libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap/InfoTableProf.hs
ghc>   HSC2HS libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap/InfoTable.hs
ghc>   HSC2HS libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap/Utils.hs
ghc>   HSC2HS libraries/ghci/dist-boot/build/GHCi/InfoTable.hs
ghc>   HSC2HS libraries/ghci/dist-boot/build/GHCi/FFI.hs
...
ghc>   HSC2HS libraries/hpc/dist-boot/build/Trace/Hpc/Reflect.hs
ghc>   HSC2HS libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap/Constants.hs
ghc>   HSC2HS libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap/InfoTable/Types.hs
ghc>   HSC2HS libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap/InfoTableProf.hs
ghc>   HSC2HS libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap/InfoTable.hs
ghc>   HSC2HS libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap/Utils.hs
ghc>   HSC2HS libraries/ghci/dist-boot/build/GHCi/InfoTable.hs
ghc>   HSC2HS libraries/ghci/dist-boot/build/GHCi/FFI.hs
...
ghc> ghc.mk:100: *** Make has restarted itself 2 times; is there a makefile bug? See https://gitlab.haskell.org/ghc/ghc/wikis/building/troubleshooting#make-has-restarted-itself-3-times-is-there-a-makefile-bug for details.  Stop.
ghc> make: *** [Makefile:126: all] Error 2 shuffle=1664105902

Looks simple, right? No, it does not. ghc’s build system detected infinite rebuild loop and bailed out. Note how Reflect.hs gets generated at elast twice with HSC2HS haskell code generator.

To explain it’s mechanics I’ll build a contrived example:

foo:
	touch foo

%.d: %.c
	echo "foo.d: foo.c"  > foo.d
	echo "foo: foo.d foo.c" >> foo.d

%.c:
	touch $*.c

-include foo.d

.PRECIOUS: foo.c

Here we dynamically generate a part of a Makefile by generating foo.d file and by including it via -include foo.d. Leading minus(-) ignores some error conditions when including files.

Let’s try it:

$ rm -f foo* && make-4.3.90
touch foo.c
echo "foo.d: foo.c"  > foo.d
echo "foo: foo.d foo.c" >> foo.d
touch foo

Note that initially foo does not contain any dependencies. GNU make has to build foo.d part first to see the rest of the dependencies.

Interestingly GNU make has to re-execute itself after foo.d is availble. We can see it in debug (-d) mode by looking up Re-executing lines:

$ rm -f foo* && LANG=C make-4.3.90 -d |& grep Re-
Re-executing[1]: make -d

Now let’s extend our foo.c rule (foo.d’s dependency) to include an unrelated and non-existent foo.h file as an output target:

--- a/makefile
+++ b/makefile
@@ -5,8 +5,9 @@ foo:
 	echo "foo.d: foo.c"  > foo.d
 	echo "foo: foo.d foo.c" >> foo.d

-%.c:
+%.c %.h:
 	touch $*.c
+	# missing 'touch $*.h'

 -include foo.d

Running:

$ rm -f foo* && make-4.3.90
touch foo.c
# missing 'touch foo.h'
echo "foo.d: foo.c"  > foo.d
echo "foo: foo.d foo.c" >> foo.d

touch foo.c
# missing 'touch foo.h'
echo "foo.d: foo.c"  > foo.d
echo "foo: foo.d foo.c" >> foo.d

touch foo.c
# missing 'touch foo.h'
echo "foo.d: foo.c"  > foo.d
echo "foo: foo.d foo.c" >> foo.d

touch foo.c
# missing 'touch foo.h'
echo "foo.d: foo.c"  > foo.d
echo "foo: foo.d foo.c" >> foo.d

touch foo.c
# missing 'touch foo.h'
echo "foo.d: foo.c"  > foo.d
echo "foo: foo.d foo.c" >> foo.d
...

GNU make fell into an infinite loop. Here missing foo.h file triggers make to always regenerate foo.d on each re-execution. Regenrated foo.d requires another re-execution. We get the loop. Previous make-4.3 version did not exhibit this behaviour:

$ rm -f foo* && make-4.3
touch foo.c
# missing 'touch foo.h'
echo "foo.d: foo.c"  > foo.d
echo "foo: foo.d foo.c" >> foo.d
touch foo

Now back to ghc. It took me some time to read through make -d output to find the offending rule. The following fix was enough to fix ghc:

--- a/rules/hs-suffix-rules-srcdir.mk
+++ b/rules/hs-suffix-rules-srcdir.mk
@@ -33,9 +33,12 @@ $1/$2/build/%.hs : $1/$2/build/%.y | $$$$(dir $$$$@)/.
 $1/$2/build/%.hs : $1/$3/%.x | $$$$(dir $$$$@)/.
 	$$(call cmd,ALEX) $$($1_$2_ALL_ALEX_OPTS) $$< -o $$@

-$1/$2/build/%_hsc.c $1/$2/build/%_hsc.h $1/$2/build/%.hs : $1/$3/%.hsc $$$$(hsc2hs_INPLACE) | $$$$(dir $$$$@)/.
+$1/$2/build/%.hs : $1/$3/%.hsc $$$$(hsc2hs_INPLACE) | $$$$(dir $$$$@)/.
 	$$(call cmd,hsc2hs_INPLACE) $$($1_$2_ALL_HSC2HS_OPTS) $$< -o $$@

+$1/$2/build/%_hsc.c $1/$2/build/%_hsc.h: $1/$2/build/%.hs
+		: # rely on previous rule to build targets
+
 # Now the rules for hs-boot files.

 $1/$2/build/%.hs-boot : $1/$3/%.hs-boot | $$$$(dir $$$$@)/.

hsc2hs does not always emit C stub part. The fix is almost identical to opensp case: we split out optional output into a separate rule.

As ghc dropped GNU make-based build system I did not try to upstream the change. Downstreams would have to carry something similar for older ghc versions they ship.

dtc case

dtc also happens to use GNU make-based build system. It’s Makefile is a lot smaller than ghc’s one. The symptom was very similar to our contrived example:

$ make-4.3.90
...
        CHK version_gen.h
         BISON dtc-parser.tab.h
         DEP dtc-lexer.lex.c
         DEP dtc-parser.tab.c
        CHK version_gen.h
         BISON dtc-parser.tab.h
         DEP dtc-lexer.lex.c
         DEP dtc-parser.tab.c
        CHK version_gen.h
         BISON dtc-parser.tab.h
         DEP dtc-lexer.lex.c
         DEP dtc-parser.tab.c
        CHK version_gen.h
         BISON dtc-parser.tab.h
         DEP dtc-lexer.lex.c
         DEP dtc-parser.tab.c
...

It took me a few hours to notice that dtc build was stuck.

The cause of cycle was again make re-execution caused by a missing file in bison rule with multiple targets. bison rule contained output that is never used by anything. The fix is trivial:

--- a/Makefile
+++ b/Makefile
@@ -384,4 +384,4 @@ clean: libfdt_clean pylibfdt_clean tests_clean
-%.tab.c %.tab.h %.output: %.y
+%.tab.c %.tab.h: %.y
 	@$(VECHO) BISON $@
 	$(BISON) -b $(basename $(basename $@)) -d $<

While at it I added a guard against infinite re-execution similar to ghc’s guard:

--- a/Makefile
+++ b/Makefile
@@ -389,3 +389,3 @@ clean: libfdt_clean pylibfdt_clean tests_clean
+ifeq ($(MAKE_RESTARTS),10)
+$(error "Make re-executed itself $(MAKE_RESTARTS) times. Infinite recursion?")
+endif

GNU make provides $(MAKE_RESTARTS) variable to detect make restarts.

Both fixes are proposed upatream as https://github.com/dgibson/dtc/pull/73.

Parting words

Rules with multiple targets are tricky and fun. GNU make-4.4 will be a bit more eager at rebuilding all of the targets if prerequisite changes. This will expose bugs in a few programs. They shoud be easy to adapt. Otherwise keeping an older version of GNU make in parallel to the newer one should be a reasonable workaround as well.

So far only opensp, ghc and dtc needed fixing.

Have fun!