Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically detecting compiler & linker used by kernel #298

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
+209 −43
Open

Automatically detecting compiler & linker used by kernel #298

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a4d904f
Automatically detecting CC & LD used by kernel
xuzhen Jan 24, 2023
111ae83
Fix test script for kernel compiled by clang
xuzhen Jan 24, 2023
0841cb0
Add clang test workflow
xuzhen Jan 25, 2023
7f44fd9
Fix printf errors caused by leading zeros
xuzhen Feb 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,3 +141,61 @@ jobs:

- name: Remove the test module
run: sudo dkms remove --all -m dkms_test -v 1.0

test-vm-clang:
name: Test in Ubuntu VM with clang
strategy:
matrix:
version:
- 22.04
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add 20.04 into the mix as well. How long are the run times?

runs-on: ubuntu-${{ matrix.version }}

steps:
- uses: actions/checkout@v3

- name: Install dependencies
run: |
sudo apt update -q
sudo apt install -qqy make libc6-dev llvm clang lld llvm-11 clang-11 lld-11 llvm-14 clang-14 lld-14 flex bison libssl-dev openssl

- name: Remove apport
run: |
sudo apt remove -qqy apport

- name: Build tiny kernels with clang
run: |
sudo apt install linux-source-5.19.0
tar -C /tmp -xf /usr/src/linux-source-5.19.0.tar.bz2
cd /tmp/linux-source-5.19.0
# make `make clean` happy
mkdir ubuntu/hio
touch ubuntu/hio/Makefile
# LLVM=-version is supported since 5.18
make LLVM=-11 tinyconfig
echo "CONFIG_MODULES=y" >> .config
echo "CONFIG_MODULE_SIG=y" >> .config
echo "CONFIG_MODULE_SIG_SHA1=y" >> .config
echo "CONFIG_MODULE_SIG_HASH=\"sha1\"" >> .config
sed -i 's/^CONFIG_LOCALVERSION=.*$/CONFIG_LOCALVERSION="-clang11"/' .config
yes $'\n' | make LLVM=-11
sudo make modules_install
sudo make install
make clean
make LLVM=-14 tinyconfig
echo "CONFIG_MODULES=y" >> .config
echo "CONFIG_MODULE_SIG=y" >> .config
echo "CONFIG_MODULE_SIG_SHA512=y" >> .config
echo "CONFIG_MODULE_SIG_HASH=\"sha512\"" >> .config
sed -i 's/^CONFIG_LOCALVERSION=.*$/CONFIG_LOCALVERSION="-clang14"/' .config
yes $'\n' | make LLVM=-14
sudo make modules_install
sudo make install
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's move this to a .github/workflow/tiny-clang.sh script and use a simple loop. Something like the following (untested)

#!/bin/bash

declare -A llvm_sha={
 [11]="1",
 [14]="512",
)

for ver in "${!llvm_sha[@]}"; do
        sha=${llvm_sha[${ver}]

        make clean
        make LLVM=-${ver} tinyconfig
        echo "CONFIG_MODULES=y" >> .config
        echo "CONFIG_MODULE_SIG=y" >> .config
        echo "CONFIG_MODULE_SIG_SHA${sha}=y" >> .config
        echo "CONFIG_MODULE_SIG_HASH=\"sha${sha}\"" >> .config
        sed -i 's/^CONFIG_LOCALVERSION=.*$/CONFIG_LOCALVERSION="-clang${ver}"/' .config
        yes $'\n' | make LLVM=-${ver}
        sudo make modules_install
        sudo make install
done


- name: Install dkms
run: sudo make install

- name: Run tests
run: |
KERNEL_VER="$(cd /tmp/linux-source-5.19.0 && make kernelversion)"
sudo env KERNEL_VER="${KERNEL_VER}-clang11" ./run_test.sh
sudo env KERNEL_VER="${KERNEL_VER}-clang14" ./run_test.sh
163 changes: 127 additions & 36 deletions dkms.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -593,23 +593,6 @@ read_conf()
[[ ! $make_command ]] && make_command="make -C $kernel_source_dir M=$dkms_tree/$module/$module_version/build"
[[ ! $clean ]] && clean="make -C $kernel_source_dir M=$dkms_tree/$module/$module_version/build clean"

# Check if clang was used to compile or lld was used to link the kernel.
if [[ -e $kernel_source_dir/vmlinux ]]; then
if readelf -p .comment $kernel_source_dir/vmlinux | grep -q clang; then
make_command="${make_command} CC=clang"
fi
if readelf -p .comment $kernel_source_dir/vmlinux | grep -q LLD; then
make_command="${make_command} LD=ld.lld"
fi
elif [[ -e "${kernel_config}" ]]; then
if grep -q CONFIG_CC_IS_CLANG=y "${kernel_config}"; then
make_command="${make_command} CC=clang"
fi
if grep -q CONFIG_LD_IS_LLD=y "${kernel_config}"; then
make_command="${make_command} LD=ld.lld"
fi
fi

# Set patch_array (including kernel specific patches)
count=0
for ((index=0; index < ${#PATCH[@]}; index++)); do
Expand Down Expand Up @@ -833,6 +816,90 @@ run_build_script() {
fi
}

# helper functions for getting cc/ld used by kernel
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mind adding a bit more context here;
Create an array of candidates, first one having the complete version suffixed and strip numbers until we find a match.

detect_toolchain_version() {
# $1 = program name, e.g. gcc, ld.lld
# $2 = major version
# $3 = minor version
# $4 = patch version, optional
local -a filenames
if [[ "$4" ]]; then
filenames+=( "$1-$((10#$2)).$((10#$3)).$((10#$4))" )
fi
filenames+=(
"$1-$((10#$2)).$((10#$3))"
"$1-$((10#$2))"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please squash this with the respective commit and add an inline comment or two.
Something like "Zero extend (or zero prefix perhaps?) the respective versions to avoid issues like XXX being detected as YYY"

"$1"
)
for f in "${filenames[@]}"; do
if command -V "$f" &>/dev/null; then
echo "$f"
return 0
fi
done
return 1
}
detect_toolchain_from_text() {
# $1 = program version text, contains x.y.z
# $2 = program name, e.g. gcc, clang
local patterns=(
's/^.*\b([0-9]+\.[0-9]+\.[0-9]+).*$/\1/p'
)
local -a ver
for p in "${patterns[@]}"; do
readarray -d '.' -t ver <<< "$(echo "$1" | sed -En "$p")"
if [[ "$ver" ]]; then
if detect_toolchain_version "$2" ${ver[0]} ${ver[1]} ${ver[2]}; then
return 0
fi
fi
done
return 1
Copy link
Collaborator

@evelikov evelikov Mar 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to read the binary, if the KCONFIG is always available?

Edit: since we always add a KCONFIG in the tests, the codepath is not covered in CI. So let's drop this, if the existing code does not work then we can worry about it.

}
detect_toolchain_from_kconfig() {
# $1 = program name in kconfig, e.g. GCC, CLANG, LD, LLD
# $2 = program name, e.g. gcc, clang, ld.bfd, ld.lld
local ver_full="$(grep "^CONFIG_$1_VERSION=" "${kernel_config}" | cut -f2 -d= )"
if [[ -z "${ver_full}" ]] || [[ "${ver_full}" = 0 ]]; then
return 1
fi
local ver_major="${ver_full:0: -4}"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a couple of examples for illustration purposes - one GCC and one CLANG. For example my system has CONFIG_GCC_VERSION=60300
Looking at the code below - seems like ver_patch is always set so one can drop the explicit check if [[ "$4" ]] in detect_toolchain_version, right?

local ver_minor="${ver_full: -4:2}"
local ver_patch="${ver_full: -2}"
detect_toolchain_version "$2" "${ver_major}" "${ver_minor}" "${ver_patch}"
return $?
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The trailing return $? should not be needed.

}
detect_toolchain_from_cc_version_text() {
# since kernel 5.8
local text="$(grep "^CONFIG_CC_VERSION_TEXT=" "${kernel_config}" | cut -f2 -d= )"
if [[ -z "$text" ]]; then
return 1
fi
if echo "$text" | grep -q gcc; then
detect_toolchain_from_text "$text" "gcc"
return $?
elif echo "$text" | grep -q clang; then
detect_toolchain_from_text "$text" "clang"
return $?
fi
return 1
}
detect_toolchain_from_vmlinux() {
# $1 = string in vmlinux .comment section, e.g. clang, LLD
# $2 = program name, e.g. clang, ld.lld
local vmlinux="${kernel_source_dir}/vmlinux"
if [[ ! -f "$vmlinux" ]]; then
return 1
fi
local text="$(readelf -p .comment "$vmlinux" 2>/dev/null | grep "$1" 2>/dev/null)"
if [[ -z "$text" ]]; then
return 1
fi
detect_toolchain_from_text "$text" "$2"
return $?
}


# Register a DKMS-ified source tree with DKMS.
# This function is smart enough to register the module if we
# passed a source tree or a tarball instead of relying on the source tree
Expand Down Expand Up @@ -1040,22 +1107,52 @@ prepare_build()
$"Check $build_dir for more information."
done

if [[ -e "${kernel_config}" ]]; then
if grep -q 'CONFIG_CC_IS_CLANG=y' "${kernel_config}"; then
local cc=clang
if command -v "$cc" >/dev/null; then
export CC="$cc"
export KERNEL_CC="$cc"
# Detect CC and LD used by kernel
if ! grep -qE '(^|\s)CC=' <<< "${make_command}"; then
# No CC specified within MAKE in dkms.conf
if [[ -e "${kernel_config}" ]]; then
if grep -q '^CONFIG_CC_IS_GCC=y' "${kernel_config}"; then
# Since kernel 4.18
KERNEL_CC="$(detect_toolchain_from_kconfig GCC gcc)"
elif grep -q '^CONFIG_CC_IS_CLANG=y' "${kernel_config}"; then
# Since kernel 4.18
KERNEL_CC="$(detect_toolchain_from_kconfig CLANG clang)"
elif [[ -f "$kernel_source_dir/.kernelvariables" ]]; then
# Debian specific
KERNEL_CC="$(echo -e "show-%:\n\t@echo \$(\$*)\ninclude $kernel_source_dir/.kernelvariables" | make -f - show-CC)"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my testing, detect_toolchain_from_kconfig works correctly and this kernelvariables workaround could be dropped. Not a big deal if you want to keep it, but in practise it's untested and never used.

else
KERNEL_CC="$(detect_toolchain_from_cc_version_text)"
fi
fi

if grep -q 'CONFIG_LD_IS_LLD=y' "${kernel_config}"; then
local ld=ld.lld
if command -v "$ld" >/dev/null; then
export LD="$ld"
export KERNEL_LD="$ld"
if [[ -z "$KERNEL_CC" ]]; then
KERNEL_CC="$(detect_toolchain_from_vmlinux GCC gcc || \
detect_toolchain_from_vmlinux clang clang)"
fi
if [[ "$KERNEL_CC" ]]; then
Copy link
Collaborator

@evelikov evelikov Mar 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since KERNEL_CC and KERNEL_LD are local, so let's use local KERNEL_CC... in this function.

Both variables were exported with #294, although searching through the Debian patches/tooling does not indicate why. So I suspect keeping them local is fine.

Wires got shorted - zfs is the one needing the exports. Although it can/will use CC/LD, which we explicit export for that purpose.

make_command="${make_command} CC=${KERNEL_CC}"
# the pre_build script may need these, e.g. zfs-dkms
export CC="${KERNEL_CC}" KERNEL_CC
fi
fi
if ! grep -qE '(^|\s)LD=' <<< "${make_command}"; then
# No LD specified within MAKE in dkms.conf
if [[ -e "${kernel_config}" ]]; then
if grep -q '^CONFIG_LD_IS_LLD=y' "${kernel_config}"; then
# Since kernel 5.8
KERNEL_LD="$(detect_toolchain_from_kconfig LLD ld.lld)"
elif grep -q '^CONFIG_LD_IS_BFD=y' "${kernel_config}"; then
# Since kernel 5.14
KERNEL_LD="$(detect_toolchain_from_kconfig LD ld.bfd)"
fi
fi
if [[ -z "$KERNEL_LD" ]]; then
KERNEL_LD="$(detect_toolchain_from_vmlinux LLD ld.lld)"
fi
if [[ "$KERNEL_LD" ]]; then
make_command="${make_command} LD=${KERNEL_LD}"
# the pre_build script may need these, e.g. zfs-dkms
export LD="${KERNEL_LD}" KERNEL_LD
fi
fi

# Run the pre_build script
Expand All @@ -1073,12 +1170,6 @@ actual_build()
echo $""
echo $"Building module:"

if [ -f "$kernel_source_dir/.kernelvariables" ]; then
export CC=$(echo -e "show-%:\n\t@echo \$(\$*)\ninclude $kernel_source_dir/.kernelvariables" | make -f - show-CC)
else
unset CC
fi

invoke_command "$clean" "Cleaning build area" background
echo $"DKMS make.log for $module-$module_version for kernel $kernelver ($arch)" >> "$build_log"
date >> "$build_log"
Expand Down Expand Up @@ -2304,7 +2395,7 @@ PATH="$PATH:/usr/lib/dkms"
umask 022

# Unset environment variables that may interfere with the build
unset CC CXX CFLAGS CXXFLAGS LDFLAGS
unset CC CXX LD CFLAGS CXXFLAGS LDFLAGS KERNEL_CC KERNEL_LD
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICT KERNEL_CC KERNEL_LD are only local, so I would drop them from here.


# Set important variables
current_kernel=$(uname -r)
Expand Down
31 changes: 24 additions & 7 deletions run_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ cd "$(dirname -- "$0")"
KERNEL_VER="${KERNEL_VER:-$(uname -r)}"
echo "Using kernel ${KERNEL_VER}"

case "$(uname -s)" in
Linux) KCONFIG="/lib/modules/${KERNEL_VER}/build/.config" ;;
GNU/kFreeBSD) KCONFIG="/usr/src/kfreebsd-headers-${KERNEL_VER}/sys/.config" ;;
esac

# Override PATH to use the local dkms binary
PATH="$(pwd):$PATH"
export PATH
Expand Down Expand Up @@ -159,6 +164,9 @@ genericize_expected_output() {
if [[ $# -ge 2 && "$2" =~ uninstall|unbuild|remove ]] ; then
sed -i '/^depmod\.\.\.$/d' ${output_log}
fi
# Remove "CC=... LD=..." part from the "make ..." line. They have different values on different platforms
sed -i '/^make /s/ CC=[0-9A-Za-z_\.\-]*[0-9A-Za-z]//' ${output_log}
sed -i '/^make /s/ LD=[0-9A-Za-z_\.\-]*[0-9A-Za-z]//' ${output_log}
# Signing related output. Drop it from the output, to be more generic
sed -i '/^Sign command:/d' ${output_log}
sed -i '/^Signing key:/d' ${output_log}
Expand Down Expand Up @@ -374,7 +382,16 @@ EOF
else
ALTER_HASH="sha512"
fi
echo "CONFIG_MODULE_SIG_HASH=\"${ALTER_HASH}\"" > /tmp/dkms_test_kconfig
if [[ -f "$KCONFIG" ]]; then
cp "$KCONFIG" /tmp/dkms_test_kconfig
if grep -q '^CONFIG_MODULE_SIG_HASH=' /tmp/dkms_test_kconfig; then
sed -i "s/^CONFIG_MODULE_SIG_HASH=.*$/CONFIG_MODULE_SIG_HASH=\"${ALTER_HASH}\"/" /tmp/dkms_test_kconfig
else
echo "CONFIG_MODULE_SIG_HASH=\"${ALTER_HASH}\"" >> /tmp/dkms_test_kconfig
fi
else
echo "CONFIG_MODULE_SIG_HASH=\"${ALTER_HASH}\"" > /tmp/dkms_test_kconfig
fi
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With my MR in mind #312, I think we can split this (+ KCONFIG above) into it's own patch and simplify:

  • assume KCONFIG is always present - error otherwise
  • change, do not set CONFIG_MODULE_SIG_HASH

run_with_expected_output dkms build -k "${KERNEL_VER}" -m dkms_test -v 1.0 --config /tmp/dkms_test_kconfig --force << EOF

Building module:
Expand Down Expand Up @@ -463,11 +480,11 @@ dkms_test/1.0, ${KERNEL_VER}, $(uname -m): installed
EOF

echo 'Checking modinfo'
run_with_expected_output sh -c "modinfo /lib/modules/${KERNEL_VER}/${expected_dest_loc}/dkms_test.ko${mod_compression_ext} | head -n 4" << EOF
filename: /lib/modules/${KERNEL_VER}/${expected_dest_loc}/dkms_test.ko${mod_compression_ext}
version: 1.0
run_with_expected_output sh -c "modinfo /lib/modules/${KERNEL_VER}/${expected_dest_loc}/dkms_test.ko${mod_compression_ext} | head -n 4 | sort" << EOF
Copy link
Collaborator

@evelikov evelikov Mar 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On which platforms (distro, version, kmod version) do we need the sort? I can send a fix to kmod, if I know a bit more.

But realistically: if we need the sort, it should happen before the headm and ideally it'll be applied to all modinfo invocations.

Edit: please split it into it's own MR (alongside the other suggestions coming shortly), so we can land that ASAP.

description: A Simple dkms test module
filename: /lib/modules/${KERNEL_VER}/${expected_dest_loc}/dkms_test.ko${mod_compression_ext}
license: GPL
version: 1.0
EOF

if [[ "${NO_SIGNING_TOOL}" = 0 ]]; then
Expand Down Expand Up @@ -585,11 +602,11 @@ if ! [[ -f "/lib/modules/${KERNEL_VER}/${expected_dest_loc}/dkms_test.ko${mod_co
fi

echo 'Checking modinfo'
run_with_expected_output sh -c "modinfo /lib/modules/${KERNEL_VER}/${expected_dest_loc}/dkms_test.ko${mod_compression_ext} | head -n 4" << EOF
filename: /lib/modules/${KERNEL_VER}/${expected_dest_loc}/dkms_test.ko${mod_compression_ext}
version: 1.0
run_with_expected_output sh -c "modinfo /lib/modules/${KERNEL_VER}/${expected_dest_loc}/dkms_test.ko${mod_compression_ext} | head -n 4 | sort" << EOF
description: A Simple dkms test module
filename: /lib/modules/${KERNEL_VER}/${expected_dest_loc}/dkms_test.ko${mod_compression_ext}
license: GPL
version: 1.0
EOF

if [[ "${NO_SIGNING_TOOL}" = 0 ]]; then
Expand Down