Update makefiles for clang 14 or newer

[davidedelvento]

I'm trying these examples on clang v14 and v15 (on an Ubuntu box) and they fail with some variant of ld.lld: error: undefined symbol: __enzyme_autodiff or square.c:(.text+0x3a): undefined reference to '__enzyme_autodiff' depending on what linker I use.

For example, using regular gnu ld and making the following changes to the Makefile

$ git diff --word-diff Makefile
diff --git a/1_square/Makefile b/1_square/Makefile
index c223268..b4f278a 100644
--- a/1_square/Makefile
+++ b/1_square/Makefile
@@ -4,7 +4,8 @@ clean:
        rm -f *.o *.ll

%.o: %.c
        [-../dockerscript.sh clang-12 /host/$^-]{+clang $^+} -O3 -Xclang -load -Xclang [-/Enzyme/enzyme/build/Enzyme/ClangEnzyme-12.so-]{+../../Enzyme/enzyme/build/Enzyme/LLVMEnzyme-14.so+} -ffast-math -o [-/host/$@-]{+$@+}

I obtain:

clang square.c -O3 -Xclang -load -Xclang ../../Enzyme/enzyme/build/Enzyme/LLVMEnzyme-14.so -ffast-math -o square.o
/usr/bin/ld: /tmp/square-977b64.o: in function `main':
square.c:(.text+0x3a): undefined reference to `__enzyme_autodiff'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:7: square.o] Error 1

Adding -fpass-plugin=../../Enzyme/enzyme/build/Enzyme/LLVMEnzyme-14.so for the new pass-manager does not change anything.

The possibly most consequential change to the Makefile is probably the switch from ClangEnzyme-version.so to LLVMEnzyme-version.so. Is that the root cause to this problem? If so, how does one build the former? There is no mention of ClangEnzyme in the cmake_install.cmake of Enzyme version da97fda9aeef27e827a8803f31c40d1b90d0f4a0 (origin/jmp/dataflow-activity-analysis).

As a workaround, I can manually run each step as described at https://enzyme.mit.edu/getting_started/UsingEnzyme/ and that works:

$ clang square.c -S -emit-llvm -o input.ll -O2 -fno-vectorize -fno-slp-vectorize -fno-unroll-loops
$ opt -enable-new-pm=0 input.ll -load=../../Enzyme/enzyme/build/Enzyme/LLVMEnzyme-14.so -enzyme -o output.ll -S
$ opt output.ll -O2 -o output_opt.ll -S
$ clang output_opt.ll -o a.exe
$ ./a.exe 3.14
Gradient square(3.140000) = 6.280000

However that is good only for single-file examples. The optcommand can't be made to work for multisource, as far as I understand it, since both opt and clang -S can only deal with one source file at the time and one would need two for this purpose:

$ clang multisource.c -S -emit-llvm -o multisource.ll -fno-vectorize -fno-slp-vectorize -fno-unroll-loops
$ clang myblas.c -S -emit-llvm -o myblas.ll -fno-vectorize -fno-slp-vectorize -fno-unroll-loops
$ opt -enable-new-pm=0 myblas.ll -load=../../Enzyme/enzyme/build/Enzyme/LLVMEnzyme-14.so -enzyme -o myblas_out.ll -S
$ opt -enable-new-pm=0 multisource.ll -load=../../Enzyme/enzyme/build/Enzyme/LLVMEnzyme-14.so -enzyme -o multisource_out.ll -S
error: <unknown>:0:0: in function preprocess_dotabs double (%struct.complex*, %struct.complex*, i32): Enzyme: No reverse pass found for myblas_cabs
 at context:   %7 = call double @myblas_cabs(double %5, double %6) #7

$ opt -enable-new-pm=0 myblas.ll multisource.ll -load=../../Enzyme/enzyme/build/Enzyme/LLVMEnzyme-14.so -enzyme -o multisource+myblas_out.ll -S
opt: Too many positional arguments specified!
Can specify at most 1 positional arguments: See: opt --help

$ clang multisource.c myblas.c -S -emit-llvm -o multisource+myblas.ll -fno-vectorize -fno-slp-vectorize -fno-unroll-loops
clang: error: cannot specify -o when generating multiple output files

so how can one fix that case?

@wsmoses sorry if the question is very trivial, but I've been unable to find guidance in the existing examples and documentation, either here or on https://github.com/EnzymeAD/Enzyme

[davidedelvento]

I found the correct syntax which fixed the issue.

For multisource, Enzyme needs to be interjecting the link pass with an LTO pass (as described in more details in the ticket mentioned above). The new syntax to make that work in LLVM v16 is

-Wl,--load-pass-plugin=../../Enzyme/enzyme/build/Enzyme/LLDEnzyme-16.so

Note the absence of -mllvm=