Project

General

Profile

Kernel hacking Debian image

The OpenWrt image is an easy way to start multiple virtual instances. But these instances usually don't provide the required infrastructure to test kernel modules extensively. And it also depends on special toolchains to prepare the used tools/modules which should tested.

It is often easier to use the same operating system in the virtual environment and on the host. Only the kernel is modified here to provide the necessary helpers for in-kernel development.

An interested reader might even extend this further to only provide a modified kernel and use the currently running rootfs also in the virtual environment. Such an approach is used in hostap's test vm but it is out of scope for this document.

Create an Image

The debian root filesystem is used here to a minimal system to boot and run the test programs. It is a simple ext4 filesystem with only userspace components from Debian. The configuration is changed to:

  • automatically mount the shared folder
  • automatically set up a static IPv4 address and hostname on bootup
  • start a test-init.sh script from the shared folder on bootup
  • disable root password
  • prefer batctl binary from shared folder's batctl subdirectory instead of virtual environment binary

The installation is also cleaned up at the end to reduce the required storage space

qemu-img create debian.img 8G
sudo mkfs.ext4 -O '^has_journal' -F debian.img
sudo mkdir debian
sudo mount -o loop debian.img debian
sudo debootstrap buster debian
sudo chroot debian apt update
sudo chroot debian apt install --no-install-recommends build-essential vim openssh-server less \
 pkg-config libnl-3-dev libnl-genl-3-dev libcap-dev tcpdump \
 trace-cmd flex bison libelf-dev libdw-dev binutils-dev libunwind-dev libssl-dev libslang2-dev liblzma-dev libperl-dev

sudo mkdir debian/root/.ssh/
ssh-add -L | sudo tee debian/root/.ssh/authorized_keys

sudo mkdir debian/host
sudo sh -c 'cat > debian/etc/fstab  << EOF
host            /host   9p      trans=virtio,version=9p2000.L 0 0
EOF'

sudo sh -c 'cat > debian/etc/rc.local << "EOF" 
#!/bin/sh -e

MAC_PART="$(ip link show enp0s1 | awk "/ether/ {print \$2}"| sed -e "s/.*://" -e "s/[\\n\\ ].*//"|awk "{print (\"0x\"\$1)*1 }")" 
IP_PART="$(echo $MAC_PART|awk "{ print \$1+50 }")" 
NODE_NR="$(echo $MAC_PART|awk "{ printf(\"%02d\", \$1) }")" 
ip addr add 192.168.251.${IP_PART}/24 dev enp0s1
ip link set up dev enp0s1
hostname "node"$NODE_NR
ip link set up dev lo
[ ! -x /host/test-init.sh ] || /host/test-init.sh
exit 0
EOF'
sudo chmod a+x debian/etc/rc.local

sudo sed -i 's/^root:[^:]*:/root::/' debian/etc/shadow

## optionally: allow ssh logins without passwords
# sudo sed -i 's/^#PermitRootLogin.*/PermitRootLogin yes/' debian/etc/ssh/sshd_config
# sudo sed -i 's/^#PermitEmptyPasswords.*/PermitEmptyPasswords yes/' debian/etc/ssh/sshd_config
# sudo sed -i 's/^UsePAM.*/UsePAM no/' debian/etc/ssh/sshd_config

## optionally: enable autologin for user root
#sudo mkdir debian/etc/systemd/system/serial-getty@hvc0.service.d/
#sudo sh -c 'cat > debian/etc/systemd/system/serial-getty@hvc0.service.d/autologin.conf  << EOF
#[Service]
#ExecStart=
#ExecStart=-/sbin/agetty --autologin root -s %I 115200,38400,9600 vt102
#EOF'

sudo sh -c 'echo '\''PATH="/host/batctl/:$PATH"'\'' >> debian/etc/profile'
sudo rm debian/var/cache/apt/archives/*.deb
sudo rm debian/var/lib/apt/lists/*
sudo e4defrag -v debian/
sudo umount debian
sudo zerofree -v debian.img

## optionally: convert image to qcow2
#sudo qemu-img convert -c -f raw -O qcow2 debian.img debian.qcow2
#sudo mv debian.qcow2 debian.img

Kernel compile

Any recent kernel can be used for the setup. We will use linux-next here to get the most recent development kernels. It is also assumed that the sources are copied to the same directory as the debian.img and a x86_64 image will be used.

The kernel will be build to enhance the virtualization and debugging experience. It is configured with:

  • basic kernel features
  • support for necessary drivers
  • kernel hacking helpers
  • kernel address + undefined sanitizers
  • support for hwsim
# make sure that libelf-dev is installed or module build will fail with something like "No rule to make target 'net/batman-adv/bat_algo.o'" 

git clone git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
cd linux-next

make allnoconfig
cat >> .config << EOF

# small configuration
CONFIG_SMP=y
CONFIG_EMBEDDED=n
# CONFIG_EXPERT is not set
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_64BIT=y
CONFIG_X86_VSYSCALL_EMULATION=n
CONFIG_IA32_EMULATION=n
CONFIG_VOP_BUS=y
CONFIG_VOP=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_VIRTIO_MENU=y
CONFIG_SCSI_VIRTIO=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_VIRTIO_INPUT=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_BLK_SCSI=y
CONFIG_VIRTIO_INPUT=y
CONFIG_VIRTIO_MMIO=y
CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
CONFIG_RPMSG_VIRTIO=y
CONFIG_VSOCKETS=y
CONFIG_VIRTIO_VSOCKETS=y
CONFIG_DRM=y
CONFIG_DRM_VIRTIO_GPU=y
CONFIG_CAIF=y
CONFIG_CAIF_VIRTIO=y
CONFIG_CRYPTO_DEV_VIRTIO=y
CONFIG_FUSE_FS=y
CONFIG_VIRTIO_FS=y
CONFIG_IOMMU_SUPPORT=y
CONFIG_VIRTIO_IOMMU=y
CONFIG_LIBNVDIMM=y
CONFIG_VIRTIO_PMEM=y
CONFIG_CRC16=y
CONFIG_LIBCRC32C=y
CONFIG_CRYPTO_SHA512=y
CONFIG_NET=y
CONFIG_INET=y
CONFIG_DEBUG_FS=y
CONFIG_IPV6=y
CONFIG_BRIDGE=y
CONFIG_VLAN_8021Q=y
CONFIG_WIRELESS=n
CONFIG_NET_9P=y
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_9P_FS=y
CONFIG_9P_FS_POSIX_ACL=y
CONFIG_9P_FS_SECURITY=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT23=y
CONFIG_TTY=y
CONFIG_HW_RANDOM=y
CONFIG_VHOST_RING=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_SCSI=y
CONFIG_NETDEVICES=y
CONFIG_NET_CORE=y
CONFIG_DEVTMPFS=y
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PVH=y
CONFIG_PARAVIRT=y
CONFIG_PARAVIRT_TIME_ACCOUNTING=y
CONFIG_PARAVIRT_SPINLOCKS=y
CONFIG_KVM_GUEST=y
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_SCRIPT=y
CONFIG_BINFMT_MISC=y
CONFIG_PCI=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_CROSS_MEMORY_ATTACH=y
CONFIG_UNIX=y
CONFIG_TMPFS=y
CONFIG_CGROUPS=y
CONFIG_BLK_CGROUP=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CGROUP_NET_CLASSID=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_SCHED=y
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y
CONFIG_INOTIFY_USER=y
CONFIG_FHANDLE=y
CONFIG_CFG80211=y
CONFIG_DUMMY=y
CONFIG_PACKET=y
CONFIG_VETH=y
CONFIG_IP_MULTICAST=y
CONFIG_NET_IPGRE_DEMUX=y
CONFIG_NET_IP_TUNNEL=y
CONFIG_NET_IPGRE=y
CONFIG_NET_IPGRE_BROADCAST=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_NO_HZ_IDLE=y
CONFIG_CPU_IDLE_GOV_HALTPOLL=y
CONFIG_PVPANIC=y

# makes boot a lot slower but required for shutdown
CONFIG_ACPI=y

#debug stuff
CONFIG_CC_STACKPROTECTOR_STRONG=y
CONFIG_LOCKUP_DETECTOR=y
CONFIG_DETECT_HUNG_TASK=y
CONFIG_SCHED_STACK_END_CHECK=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_PROVE_LOCKING=y
CONFIG_LOCK_STAT=y
CONFIG_DEBUG_LOCKDEP=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_PI_LIST=y
CONFIG_DEBUG_SG=y
CONFIG_DEBUG_NOTIFIERS=y
CONFIG_PROVE_RCU_REPEATEDLY=y
CONFIG_SPARSE_RCU_POINTER=y
CONFIG_DEBUG_STRICT_USER_COPY_CHECKS=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_DEBUG_RODATA=y
CONFIG_DEBUG_RODATA_TEST=n
CONFIG_DEBUG_SET_MODULE_RONX=y
CONFIG_PAGE_EXTENSION=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_DEBUG_OBJECTS=y
CONFIG_DEBUG_OBJECTS_FREE=y
CONFIG_DEBUG_OBJECTS_TIMERS=y
CONFIG_DEBUG_OBJECTS_WORK=y
CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
CONFIG_DEBUG_OBJECTS_PERCPU_COUNTER=y
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=8000
CONFIG_DEBUG_STACK_USAGE=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_GDB_SCRIPTS=y
CONFIG_READABLE_ASM=y
CONFIG_STACK_VALIDATION=y
CONFIG_WQ_WATCHDOG=y
CONFIG_DEBUG_KOBJECT_RELEASE=y
CONFIG_DEBUG_WQ_FORCE_RR_CPU=y
CONFIG_OPTIMIZE_INLINING=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_DEBUG_SECTION_MISMATCH=y
CONFIG_UNWINDER_ORC=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_TRACER_SNAPSHOT=y
CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP=y
CONFIG_STACK_TRACER=y
CONFIG_UPROBE_EVENTS=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_HIST_TRIGGERS=y

# for GCC 5+
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y
CONFIG_UBSAN_SANITIZE_ALL=y
CONFIG_UBSAN=y
CONFIG_UBSAN_NULL=y
EOF
make olddefconfig

cat >> .config << EOF
# allow to use unsigned regdb with hwsim
CONFIG_EXPERT=y
CONFIG_CFG80211_CERTIFICATION_ONUS=y
# CONFIG_CFG80211_REQUIRE_SIGNED_REGDB is not set
EOF
make olddefconfig

make all -j$(nproc || echo 1)

Build the BIOS

The (sea)bios used by qemu is nice to boot all kind of legacy images but reduces the performance for booting a paravirtualized Linux system. Something like qboot works better for this purpose:

git clone https://github.com/bonzini/qboot.git
make -C qboot

Building the batman-adv module

The kernel module can be build outside the virtual environment and shared over the 9p mount. The path to the kernel sources have to be provided to the make process

make KERNELPATH="$(pwd)/../linux-next" 

The kernel module can also be compiled in a way which creates better stack traces and increases the usability with (k)gdb:

make EXTRA_CFLAGS="-fno-inline -Og -fno-optimize-sibling-calls" KERNELPATH="$(pwd)/../linux-next" V=1

Start of the environment

virtual network initialization

The virtual-network.sh from the OpenWrt environment can be reused again.

VM instances bringup

The run.sh from the OpenWrt environment can mostly be reused. There are only minimal adjustments required.

The BASE_IMG is of course no longer the same because a new image "debian.img" was created for our new environment. The image also doesn't contain a bootloader or kernel anymore. The kernel must now be supplied manually to qemu.

BASE_IMG=debian.img
BOOTARGS+=("-bios" "qboot/bios.bin")
BOOTARGS+=("-kernel" "linux-next/arch/x86/boot/bzImage")
BOOTARGS+=("-append" "root=/dev/vda rw console=hvc0 nokaslr tsc=reliable no_timer_check noreplace-smp rootfstype=ext4 rcupdate.rcu_expedited=1 reboot=t pci=lastbus=0 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1")
BOOTARGS+=("-device" "virtconsole,chardev=charconsole0,id=console0")

It is also recommended to use linux-next/vmlinux instead of bzImage with qemu 4.0.0 (or later)

Automatic test initialization

The test-init.sh from the OpenWrt environment is always test specific. But its main functionality is still the same as before. A simple example would be:

cat > test-init.sh << "EOF" 
#! /bin/sh

set -e

## get internet access
dhclient enp0s2

## Simple batman-adv setup

# ip link add dummy0 type dummy
ip link set up dummy0

rmmod batman-adv || true
insmod /host/batman-adv/net/batman-adv/batman-adv.ko
/host/batctl/batctl routing_algo BATMAN_IV
/host/batctl/batctl if add dummy0
/host/batctl/batctl it 5000
/host/batctl/batctl if add enp0s1
ip link set up dev enp0s1
ip link set up dev bat0
EOF

chmod +x test-init.sh

Start

The startup method from the OpenWrt environment should be used here.