kern_error_monitor

BPF-based monitor for kernel error messages (KERN_ERR level). Captures messages from printk(KERN_ERR ...), pr_err(), and dev_err() in real time and displays them with fully formatted arguments (like dmesg).

How it works

The monitor uses a hybrid approach combining BPF with /dev/kmsg:

1. BPF kprobe on vprintk_emit() — detects KERN_ERR messages and captures metadata (PID, process name, device info) that /dev/kmsg alone does not provide.

2. /dev/kmsg reader — reads the fully formatted message text after the kernel stores it. This gives actual values (e.g., code=-12) instead of format specifiers (code=%d).

3. Timestamp correlation — BPF events fire at vprintk_emit entry (before formatting), and /dev/kmsg entries appear after. The daemon matches them using boot-relative timestamps from both sources.

KERN_ERR detection

The BPF program detects KERN_ERR in two ways:

• Explicit level parameter — dev_err() and /dev/kmsg writes pass level=3 directly to vprintk_emit().
• Format string prefix — printk(KERN_ERR ...) and pr_err() encode the level as \001 + 3 at the start of the format string.

Prerequisites

sudo apt install clang gcc libbpf-dev libelf-dev zlib1g-dev \
    linux-tools-common linux-headers-$(uname -r)

Verify BTF is available (required for vmlinux.h generation):

ls /sys/kernel/btf/vmlinux

Build

make

This will:

1. Generate vmlinux.h from the running kernel's BTF
2. Compile the BPF program (kern_error_monitor.bpf.o)
3. Generate the BPF skeleton header (kern_error_monitor.skel.h)
4. Compile the userspace daemon (kern_error_monitor)

To clean build artifacts:

make clean

Running the monitor

The daemon requires root privileges to load the BPF program and read /dev/kmsg:

sudo ./kern_error_monitor

Output (dmesg-style with PID/comm from BPF):

Monitoring kernel error messages (KERN_ERR)...
Press Ctrl-C to exit.

[19073.456409] test_kern_err: [TEST 1] printk KERN_ERR message                  (pid=1234 insmod)
[19073.456411] test_kern_err: [TEST 2] pr_err message, code=-12                 (pid=1234 insmod)
[19073.456413] test_kern_err: [TEST 3] simulated device failure on irq 42       (pid=1234 insmod)

Press Ctrl-C to stop.

Testing

Test 1: Kernel module (recommended)

Build and load the test module, which generates printk(KERN_ERR ...) and pr_err() messages with format arguments:

# Build the test module
cd test_module
make
cd ..

# Terminal 1: start the monitor
sudo ./kern_error_monitor

# Terminal 2: load the test module
sudo insmod test_module/test_kern_err.ko

# You should see 3 KERN_ERR events with actual values (not %d).
# The pr_warn and pr_info messages should NOT appear.

# Unload the test module when done
sudo rmmod test_kern_err

Expected output (3 events with formatted values):

[19073.456409] test_kern_err: [TEST 1] printk KERN_ERR message                  (pid=1234 insmod)
[19073.456411] test_kern_err: [TEST 2] pr_err message, code=-12                 (pid=1234 insmod)
[19073.456413] test_kern_err: [TEST 3] simulated device failure on irq 42       (pid=1234 insmod)

The pr_warn and pr_info messages from the module should not appear because they are not KERN_ERR level.

Test 2: /dev/kmsg (no module needed)

Write an error-level message directly to the kernel log:

# Terminal 1: start the monitor
sudo ./kern_error_monitor

# Terminal 2: write a KERN_ERR message
echo "<3>kern_error_monitor test: hello from /dev/kmsg" | sudo tee /dev/kmsg

Test 3: Trigger real kernel errors

Any real kernel error will be captured. Examples:

# Try to mount a non-existent device (may produce KERN_ERR on some systems)
sudo mount /dev/sdZZZ /mnt 2>/dev/null

# Trigger a USB error by unplugging a device while in use

Troubleshooting

"Failed to load BPF skeleton": Ensure BTF is enabled in your kernel (CONFIG_DEBUG_INFO_BTF=y). Check with:

zcat /proc/config.gz 2>/dev/null | grep CONFIG_DEBUG_INFO_BTF || \
grep CONFIG_DEBUG_INFO_BTF /boot/config-$(uname -r)

"Failed to attach BPF skeleton": Verify vprintk_emit is available:

sudo grep vprintk_emit /proc/kallsyms

"Failed to open /dev/kmsg": The daemon must run as root (or with CAP_SYSLOG in addition to CAP_BPF + CAP_PERFMON).

Permission denied: The daemon must run as root (or with CAP_BPF + CAP_PERFMON + CAP_SYSLOG capabilities).

Files

File	Description
kern_error_monitor.h	Shared event structure (BPF + userspace)
kern_error_monitor.bpf.c	BPF program (kprobe on vprintk_emit)
kern_error_monitor.c	Userspace daemon (BPF + /dev/kmsg hybrid)
Makefile	Build system
test_module/test_kern_err.c	Test kernel module
test_module/Makefile	Test module build