The kernel module xt_RTPENGINE caused rtpengine-recording to become unresponsive, resulting in zombie processes and ultimately leading to a complete system freeze.

[Young-LLL-6130]

rtpengine version the issue has been seen with

Version: 13.4.1.7+0~mr13.4.1.7

Used distribution and its version

Debian GNU/Linux 12 (bookworm)

Linux kernel version used

6.1.0-42-amd64

CPU architecture issue was seen on (see uname -m)

x86_64

Expected behaviour you didn't see

rtpengine-recording working fine

Unexpected behaviour you saw

system freeze, rtpengine-recording no response and has defunct process:
root@debian:/var/log# ps -ef|grep rtpengine
rtpengi+ 2280049 1 6 Feb11 ? 00:59:24 [rtpengine-recor]
rtpengi+ 2280320 1 5 Feb11 ? 00:48:43 /usr/bin/rtpengine -f --no-log-timestamps --pidfile /run/rtpengine/ngcp-rtpengine-daemon.pid --config-file /etc/rtpengine/rtpengine.conf
rtpengi+ 3562584 1 16 14:00 ? 00:04:50 /usr/bin/rtpengine-recording -f --no-log-timestamps --pidfile /run/rtpengine-recording/ngcp-rtpengine-recording-daemon.pid --config-file /etc/rtpengine/rtpengine-recording.conf

Steps to reproduce the problem

randomly

kern.log

Additional program output to the terminal or logs illustrating the issue

kern.log:

Feb 12 13:48:28 debian kernel: [153214.701047] list_del corruption. prev->next should be ffffa0399ac63a28, but was 0000000000000000. (prev=ffffa039deb9f570)
Feb 12 13:48:28 debian kernel: [153214.701156] ------------[ cut here ]------------
Feb 12 13:48:28 debian kernel: [153214.701159] kernel BUG at lib/list_debug.c:59!
Feb 12 13:48:28 debian kernel: [153214.701195] invalid opcode: 0000 [#1] PREEMPT SMP PTI
Feb 12 13:48:28 debian kernel: [153214.701230] CPU: 15 PID: 2280364 Comm: rtpengine-recor Tainted: G           OE      6.1.0-42-amd64 #1  Debian 6.1.159-1
Feb 12 13:48:28 debian kernel: [153214.701271] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 2.19.0 02/22/2024
Feb 12 13:48:28 debian kernel: [153214.701302] RIP: 0010:__list_del_entry_valid.cold+0x37/0x6f
Feb 12 13:48:28 debian kernel: [153214.701345] Code: fe ff 0f 0b 48 89 d1 48 c7 c7 78 91 fa a5 48 89 c2 e8 b1 9b fe ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 28 91 fa a5 e8 9d 9b fe ff <0f> 0b 48 89 fe 48 89 ca 48 c7 c7 f0 90 fa a5 e8 89 9b fe ff 0f 0b
Feb 12 13:48:28 debian kernel: [153214.701417] RSP: 0018:ffffc34484097bc8 EFLAGS: 00010046
Feb 12 13:48:28 debian kernel: [153214.701443] RAX: 000000000000006d RBX: ffffa0399ac63a10 RCX: 0000000000000000
Feb 12 13:48:28 debian kernel: [153214.701473] RDX: 0000000000000000 RSI: ffffa0409f9e03e0 RDI: ffffa0409f9e03e0
Feb 12 13:48:28 debian kernel: [153214.701501] RBP: ffffa039deb9f568 R08: 0000000000000000 R09: ffffc34484097a60
Feb 12 13:48:28 debian kernel: [153214.701526] R10: 0000000000000003 R11: ffffa040bff807a8 R12: 0000000000000282
Feb 12 13:48:28 debian kernel: [153214.701551] R13: ffffa039b0616e00 R14: 0000000000000000 R15: ffffa039b0616e00
Feb 12 13:48:28 debian kernel: [153214.701577] FS:  00007f34ed5f56c0(0000) GS:ffffa0409f9c0000(0000) knlGS:0000000000000000
Feb 12 13:48:28 debian kernel: [153214.701623] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 12 13:48:28 debian kernel: [153214.701645] CR2: 00007f1f0b414000 CR3: 00000001404fe002 CR4: 00000000003706e0
Feb 12 13:48:28 debian kernel: [153214.701671] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 12 13:48:28 debian kernel: [153214.701695] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Feb 12 13:48:28 debian kernel: [153214.701720] Call Trace:
Feb 12 13:48:28 debian kernel: [153214.701733]  <TASK>
Feb 12 13:48:28 debian kernel: [153214.701744]  remove_wait_queue+0x20/0x60
Feb 12 13:48:28 debian kernel: [153214.701778]  ep_remove+0x40/0x1e0
Feb 12 13:48:28 debian kernel: [153214.701811]  do_epoll_ctl+0x888/0xf50
Feb 12 13:48:28 debian kernel: [153214.701844]  ? __x64_sys_epoll_wait+0x6f/0x110
Feb 12 13:48:28 debian kernel: [153214.701880]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 12 13:48:28 debian kernel: [153214.701920]  __x64_sys_epoll_ctl+0x6c/0xa0
Feb 12 13:48:28 debian kernel: [153214.701953]  do_syscall_64+0x55/0xb0
Feb 12 13:48:28 debian kernel: [153214.701987]  ? do_futex+0xda/0x1b0
Feb 12 13:48:28 debian kernel: [153214.702018]  ? __x64_sys_futex+0x8e/0x1d0
Feb 12 13:48:28 debian kernel: [153214.702052]  ? exit_to_user_mode_prepare+0x44/0x240
Feb 12 13:48:28 debian kernel: [153214.702091]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 12 13:48:28 debian kernel: [153214.702126]  ? do_syscall_64+0x61/0xb0
Feb 12 13:48:28 debian kernel: [153214.702159]  ? __rseq_handle_notify_resume+0xa9/0x480
Feb 12 13:48:28 debian kernel: [153214.702198]  ? do_futex+0x106/0x1b0
Feb 12 13:48:28 debian kernel: [153214.702229]  ? exit_to_user_mode_prepare+0x16b/0x240
Feb 12 13:48:28 debian kernel: [153214.702268]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 12 13:48:28 debian kernel: [153214.702304]  ? do_syscall_64+0x61/0xb0
Feb 12 13:48:28 debian kernel: [153214.702336]  ? ksys_read+0xd4/0xf0
Feb 12 13:48:28 debian kernel: [153214.702369]  ? exit_to_user_mode_prepare+0x44/0x240
Feb 12 13:48:28 debian kernel: [153214.702407]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 12 13:48:28 debian kernel: [153214.702442]  ? do_syscall_64+0x61/0xb0
Feb 12 13:48:28 debian kernel: [153214.702474]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Feb 12 13:48:28 debian kernel: [153214.702514] RIP: 0033:0x7f34fe727c0a
Feb 12 13:48:28 debian kernel: [153214.703339] Code: 48 8b 0d f9 91 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 e9 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d c6 91 0c 00 f7 d8 64 89 01 48
Feb 12 13:48:28 debian kernel: [153214.705077] RSP: 002b:00007f34ed5f3f38 EFLAGS: 00000246 ORIG_RAX: 00000000000000e9
Feb 12 13:48:28 debian kernel: [153214.705821] RAX: ffffffffffffffda RBX: ffffffffffffffff RCX: 00007f34fe727c0a
Feb 12 13:48:28 debian kernel: [153214.706561] RDX: 00000000000000bf RSI: 0000000000000002 RDI: 0000000000000004
Feb 12 13:48:28 debian kernel: [153214.707312] RBP: 00007f34bc086b40 R08: 0000000000000001 R09: 00007f34bc000840
Feb 12 13:48:28 debian kernel: [153214.708055] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f34d006ca40
Feb 12 13:48:28 debian kernel: [153214.708783] R13: 0000000000000000 R14: 00007ffce922ad00 R15: 00007f34ecdf5000
Feb 12 13:48:28 debian kernel: [153214.709483]  </TASK>
Feb 12 13:48:28 debian kernel: [153214.710178] Modules linked in: xt_RTPENGINE(OE) xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_set nft_compat nf_tables libcrc32c ip_set_hash_net ip_set nfnetlink sunrpc binfmt_misc intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common sb_edac nls_ascii nls_cp437 vfat x86_pkg_temp_thermal intel_powerclamp fat ipmi_ssif coretemp kvm_intel kvm irqbypass ghash_clmulni_intel sha512_ssse3 sha512_generic sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd mgag200 rapl intel_cstate ledtrig_audio dell_smbios drm_shmem_helper dell_wmi_descriptor iTCO_wdt intel_pmc_bxt drm_kms_helper intel_uncore ipmi_si rfkill mei_me ipmi_devintf video iTCO_vendor_support evdev mei dcdbas i2c_algo_bit watchdog pcspkr mxm_wmi sg ipmi_msghandler acpi_power_meter button drm configfs loop fuse efi_pstore efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_mod sd_mod t10_pi crc64_rocksoft sr_mod crc64 crc_t10dif cdrom crct10dif_generic ahci
Feb 12 13:48:28 debian kernel: [153214.710287]  libahci ehci_pci ehci_hcd crct10dif_pclmul libata megaraid_sas tg3 crct10dif_common crc32_pclmul usbcore crc32c_intel scsi_mod libphy lpc_ich usb_common scsi_common wmi [last unloaded: xt_RTPENGINE(OE)]
Feb 12 13:48:28 debian kernel: [153214.717857] ---[ end trace 0000000000000000 ]---
Feb 12 13:48:28 debian kernel: [153214.776440] RIP: 0010:__list_del_entry_valid.cold+0x37/0x6f
Feb 12 13:48:28 debian kernel: [153214.777353] Code: fe ff 0f 0b 48 89 d1 48 c7 c7 78 91 fa a5 48 89 c2 e8 b1 9b fe ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 28 91 fa a5 e8 9d 9b fe ff <0f> 0b 48 89 fe 48 89 ca 48 c7 c7 f0 90 fa a5 e8 89 9b fe ff 0f 0b
Feb 12 13:48:28 debian kernel: [153214.779120] RSP: 0018:ffffc34484097bc8 EFLAGS: 00010046
Feb 12 13:48:28 debian kernel: [153214.780020] RAX: 000000000000006d RBX: ffffa0399ac63a10 RCX: 0000000000000000
Feb 12 13:48:28 debian kernel: [153214.780880] RDX: 0000000000000000 RSI: ffffa0409f9e03e0 RDI: ffffa0409f9e03e0
Feb 12 13:48:28 debian kernel: [153214.781761] RBP: ffffa039deb9f568 R08: 0000000000000000 R09: ffffc34484097a60
Feb 12 13:48:28 debian kernel: [153214.782632] R10: 0000000000000003 R11: ffffa040bff807a8 R12: 0000000000000282
Feb 12 13:48:28 debian kernel: [153214.783520] R13: ffffa039b0616e00 R14: 0000000000000000 R15: ffffa039b0616e00
Feb 12 13:48:28 debian kernel: [153214.784424] FS:  00007f34ed5f56c0(0000) GS:ffffa0409f9c0000(0000) knlGS:0000000000000000
Feb 12 13:48:28 debian kernel: [153214.785327] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 12 13:48:28 debian kernel: [153214.786251] CR2: 00007f1f0b414000 CR3: 00000001404fe002 CR4: 00000000003706e0
Feb 12 13:48:28 debian kernel: [153214.787177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 12 13:48:28 debian kernel: [153214.788103] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Feb 12 13:48:28 debian kernel: [153214.789025] note: rtpengine-recor[2280364] exited with irqs disabled
Feb 12 13:48:28 debian kernel: [153214.789995] note: rtpengine-recor[2280364] exited with preempt_count 1

Anything else?

No response

[Young-LLL-6130]

After the upgrade to version 13.5.1.6+0~mr13.5.1.6, the problem still occurs every day.

[rfuchs]

Without any idea how to reproduce this, this isn't really actionable

[Young-LLL-6130]

Without any idea how to reproduce this, this isn't really actionable

Whatever, thanks a lot, this is the kern.log of today if it can help, today the server freeze directly. By the way, Could this be a hardware issue? We have already tried several kernel versions(6.1.0-38-amd64 / 6.1.0-40-amd64 / 6.1.0-42-amd64).:

Feb 28 11:00:10 debian kernel: [43097.405640] perf: interrupt took too long (2508 > 2500), lowering kernel.perf_event_max_sample_rate to 79500
Feb 28 12:14:42 debian kernel: [47568.751247] perf: interrupt took too long (3136 > 3135), lowering kernel.perf_event_max_sample_rate to 63750
Feb 28 15:01:36 debian kernel: [57583.067159] perf: interrupt took too long (3923 > 3920), lowering kernel.perf_event_max_sample_rate to 50750
Feb 28 16:41:21 debian kernel: [63568.155494] list_del corruption. prev->next should be ffff9f395990e168, but was ffff9f37ee40b770. (prev=ffff9f37ee40b770)
Feb 28 16:41:21 debian kernel: [63568.155602] ------------[ cut here ]------------
Feb 28 16:41:21 debian kernel: [63568.155605] kernel BUG at lib/list_debug.c:59!
Feb 28 16:41:21 debian kernel: [63568.155642] invalid opcode: 0000 [#1] PREEMPT SMP PTI
Feb 28 16:41:21 debian kernel: [63568.155681] CPU: 24 PID: 1286 Comm: rtpengine-recor Tainted: G           OE      6.1.0-38-amd64 #1  Debian 6.1.147-1
Feb 28 16:41:21 debian kernel: [63568.155748] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 2.19.0 02/22/2024
Feb 28 16:41:21 debian kernel: [63568.155795] RIP: 0010:__list_del_entry_valid.cold+0x37/0x6f
Feb 28 16:41:21 debian kernel: [63568.155845] Code: fe ff 0f 0b 48 89 d1 48 c7 c7 10 85 da 9f 48 89 c2 e8 25 9c fe ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 c0 84 da 9f e8 11 9c fe ff <0f> 0b 48 89 fe 48 89 ca 48 c7 c7 88 84 da 9f e8 fd 9b fe ff 0f 0b
Feb 28 16:41:21 debian kernel: [63568.155956] RSP: 0018:ffffc15ec2e7bae8 EFLAGS: 00010046
Feb 28 16:41:21 debian kernel: [63568.155996] RAX: 000000000000006d RBX: ffff9f395990e150 RCX: 0000000000000000
Feb 28 16:41:21 debian kernel: [63568.156044] RDX: 0000000000000000 RSI: ffff9f3b1fc203e0 RDI: ffff9f3b1fc203e0
Feb 28 16:41:21 debian kernel: [63568.156091] RBP: ffff9f37ee40b768 R08: 0000000000000000 R09: ffffc15ec2e7b980
Feb 28 16:41:21 debian kernel: [63568.156138] R10: 0000000000000003 R11: ffff9f3b3ff807a8 R12: 0000000000000282
Feb 28 16:41:21 debian kernel: [63568.156184] R13: ffff9f32f4f26100 R14: 0000000000000000 R15: ffff9f32f4f26100
Feb 28 16:41:21 debian kernel: [63568.156229] FS:  00007ff5229e16c0(0000) GS:ffff9f3b1fc00000(0000) knlGS:0000000000000000
Feb 28 16:41:21 debian kernel: [63568.156280] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 28 16:41:21 debian kernel: [63568.156318] CR2: 00005558b106e93f CR3: 00000001121a2003 CR4: 00000000003706e0
Feb 28 16:41:21 debian kernel: [63568.156363] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 28 16:41:21 debian kernel: [63568.156407] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Feb 28 16:41:21 debian kernel: [63568.156451] Call Trace:
Feb 28 16:41:21 debian kernel: [63568.156471]  <TASK>
Feb 28 16:41:21 debian kernel: [63568.156489]  remove_wait_queue+0x20/0x60
Feb 28 16:41:21 debian kernel: [63568.156527]  ep_remove+0x40/0x1e0
Feb 28 16:41:21 debian kernel: [63568.156557]  do_epoll_ctl+0x884/0xf40
Feb 28 16:41:21 debian kernel: [63568.156588]  ? vfs_read+0xa5/0x310
Feb 28 16:41:21 debian kernel: [63568.156621]  __x64_sys_epoll_ctl+0x6c/0xa0
Feb 28 16:41:21 debian kernel: [63568.156654]  do_syscall_64+0x55/0xb0
Feb 28 16:41:21 debian kernel: [63568.156685]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 28 16:41:21 debian kernel: [63568.156720]  ? do_syscall_64+0x61/0xb0
Feb 28 16:41:21 debian kernel: [63568.156751]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 28 16:41:21 debian kernel: [63568.156784]  ? do_syscall_64+0x61/0xb0
Feb 28 16:41:21 debian kernel: [63568.156814]  ? __x64_sys_futex+0x8e/0x1d0
Feb 28 16:41:21 debian kernel: [63568.156848]  ? exit_to_user_mode_prepare+0x40/0x1e0
Feb 28 16:41:21 debian kernel: [63568.156886]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 28 16:41:21 debian kernel: [63568.156920]  ? do_syscall_64+0x61/0xb0
Feb 28 16:41:21 debian kernel: [63568.156951]  ? do_futex+0xda/0x1b0
Feb 28 16:41:21 debian kernel: [63568.156980]  ? __x64_sys_futex+0x8e/0x1d0
Feb 28 16:41:21 debian kernel: [63568.157013]  ? exit_to_user_mode_prepare+0x40/0x1e0
Feb 28 16:41:21 debian kernel: [63568.157050]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 28 16:41:21 debian kernel: [63568.157084]  ? do_syscall_64+0x61/0xb0
Feb 28 16:41:21 debian kernel: [63568.157115]  ? __rseq_handle_notify_resume+0xa9/0x4a0
Feb 28 16:41:21 debian kernel: [63568.157154]  ? do_futex+0x106/0x1b0
Feb 28 16:41:21 debian kernel: [63568.157185]  ? exit_to_user_mode_prepare+0x14b/0x1e0
Feb 28 16:41:21 debian kernel: [63568.157222]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 28 16:41:21 debian kernel: [63568.158444]  ? do_syscall_64+0x61/0xb0
Feb 28 16:41:21 debian kernel: [63568.159659]  ? do_syscall_64+0x61/0xb0
Feb 28 16:41:21 debian kernel: [63568.160861]  ? exit_to_user_mode_prepare+0x40/0x1e0
Feb 28 16:41:21 debian kernel: [63568.162031]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 28 16:41:21 debian kernel: [63568.163159]  ? do_syscall_64+0x61/0xb0
Feb 28 16:41:21 debian kernel: [63568.164282]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Feb 28 16:41:21 debian kernel: [63568.165412] RIP: 0033:0x7ff53db27c0a
Feb 28 16:41:21 debian kernel: [63568.166510] Code: 48 8b 0d f9 91 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 e9 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d c6 91 0c 00 f7 d8 64 89 01 48
Feb 28 16:41:21 debian kernel: [63568.168805] RSP: 002b:00007ff5229dff38 EFLAGS: 00000246 ORIG_RAX: 00000000000000e9
Feb 28 16:41:21 debian kernel: [63568.169947] RAX: ffffffffffffffda RBX: ffffffffffffffff RCX: 00007ff53db27c0a
Feb 28 16:41:21 debian kernel: [63568.171057] RDX: 000000000000001b RSI: 0000000000000002 RDI: 0000000000000004
Feb 28 16:41:21 debian kernel: [63568.172152] RBP: 00007ff4d81611a0 R08: 0000000000000001 R09: 00007ff4d8114400
Feb 28 16:41:21 debian kernel: [63568.173241] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ff51c07a960
Feb 28 16:41:21 debian kernel: [63568.174329] R13: 0000000000000000 R14: 00007ffe001f1b20 R15: 00007ff5221e1000
Feb 28 16:41:21 debian kernel: [63568.175408]  </TASK>
Feb 28 16:41:21 debian kernel: [63568.176456] Modules linked in: xt_RTPENGINE(OE) xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_set nft_compat nf_tables libcrc32c ip_set_hash_net ip_set nfnetlink sunrpc intel_rapl_msr intel_rapl_common binfmt_misc intel_uncore_frequency intel_uncore_frequency_common ipmi_ssif sb_edac nls_ascii x86_pkg_temp_thermal nls_cp437 intel_powerclamp vfat coretemp fat kvm_intel kvm irqbypass ghash_clmulni_intel sha512_ssse3 sha512_generic sha256_ssse3 sha1_ssse3 aesni_intel ledtrig_audio dell_smbios crypto_simd cryptd dell_wmi_descriptor mgag200 rfkill rapl drm_shmem_helper video iTCO_wdt ipmi_si drm_kms_helper intel_cstate mei_me intel_pmc_bxt iTCO_vendor_support ipmi_devintf sg mei i2c_algo_bit intel_uncore watchdog evdev pcspkr ipmi_msghandler dcdbas mxm_wmi acpi_power_meter button drm fuse configfs loop efi_pstore efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_mod sd_mod t10_pi sr_mod cdrom crc64_rocksoft crc64 crc_t10dif crct10dif_generic ahci
Feb 28 16:41:21 debian kernel: [63568.176632]  libahci ehci_pci libata ehci_hcd megaraid_sas crct10dif_pclmul tg3 crct10dif_common crc32_pclmul usbcore crc32c_intel scsi_mod libphy lpc_ich usb_common scsi_common wmi
Feb 28 16:41:21 debian kernel: [63568.189037] ---[ end trace 0000000000000000 ]---
Feb 28 16:41:21 debian kernel: [63568.248542] RIP: 0010:__list_del_entry_valid.cold+0x37/0x6f
Feb 28 16:41:21 debian kernel: [63568.249373] Code: fe ff 0f 0b 48 89 d1 48 c7 c7 10 85 da 9f 48 89 c2 e8 25 9c fe ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 c0 84 da 9f e8 11 9c fe ff <0f> 0b 48 89 fe 48 89 ca 48 c7 c7 88 84 da 9f e8 fd 9b fe ff 0f 0b
Feb 28 16:41:21 debian kernel: [63568.250928] RSP: 0018:ffffc15ec2e7bae8 EFLAGS: 00010046
Feb 28 16:41:21 debian kernel: [63568.251720] RAX: 000000000000006d RBX: ffff9f395990e150 RCX: 0000000000000000
Feb 28 16:41:21 debian kernel: [63568.252529] RDX: 0000000000000000 RSI: ffff9f3b1fc203e0 RDI: ffff9f3b1fc203e0
Feb 28 16:41:21 debian kernel: [63568.253326] RBP: ffff9f37ee40b768 R08: 0000000000000000 R09: ffffc15ec2e7b980
Feb 28 16:41:21 debian kernel: [63568.254103] R10: 0000000000000003 R11: ffff9f3b3ff807a8 R12: 0000000000000282
Feb 28 16:41:21 debian kernel: [63568.254888] R13: ffff9f32f4f26100 R14: 0000000000000000 R15: ffff9f32f4f26100
Feb 28 16:41:21 debian kernel: [63568.255677] FS:  00007ff5229e16c0(0000) GS:ffff9f3b1fc00000(0000) knlGS:0000000000000000
Feb 28 16:41:21 debian kernel: [63568.256478] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 28 16:41:21 debian kernel: [63568.257285] CR2: 00005558b106e93f CR3: 00000001121a2003 CR4: 00000000003706e0
Feb 28 16:41:21 debian kernel: [63568.258069] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 28 16:41:21 debian kernel: [63568.258857] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Feb 28 16:41:21 debian kernel: [63568.259643] note: rtpengine-recor[1286] exited with irqs disabled
Feb 28 16:41:21 debian kernel: [63568.260450] note: rtpengine-recor[1286] exited with preempt_count 1
Feb 28 16:41:45 debian kernel: [63592.732619] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Feb 28 16:41:45 debian kernel: [63592.733070] rcu: 	24-...0: (1 GPs behind) idle=9654/1/0x4000000000000000 softirq=1967982/1967983 fqs=2122
Feb 28 16:41:45 debian kernel: [63592.733454] 	(detected by 15, t=5252 jiffies, g=9161497, q=36375 ncpus=32)
Feb 28 16:41:45 debian kernel: [63592.733833] Sending NMI from CPU 15 to CPUs 24:
Feb 28 16:41:45 debian kernel: [63592.734215] NMI backtrace for cpu 24
Feb 28 16:41:45 debian kernel: [63592.734220] CPU: 24 PID: 1330 Comm: poller Tainted: G      D    OE      6.1.0-38-amd64 #1  Debian 6.1.147-1
Feb 28 16:41:45 debian kernel: [63592.734225] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 2.19.0 02/22/2024
Feb 28 16:41:45 debian kernel: [63592.734227] RIP: 0010:native_queued_spin_lock_slowpath+0x69/0x2a0
Feb 28 16:41:45 debian kernel: [63592.734239] Code: 00 77 75 f0 0f ba 2b 08 0f 92 c2 8b 03 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 77 51 85 c0 74 0e 8b 03 84 c0 74 08 f3 90 <8b> 03 84 c0 75 f8 b8 01 00 00 00 66 89 03 5b 5d 41 5c 41 5d c3 cc
Feb 28 16:41:45 debian kernel: [63592.734242] RSP: 0018:ffffc15ec2feb9b8 EFLAGS: 00000002
Feb 28 16:41:45 debian kernel: [63592.734247] RAX: 0000000000000101 RBX: ffff9f37ee40b768 RCX: 0000000000000000
Feb 28 16:41:45 debian kernel: [63592.734249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f37ee40b768
Feb 28 16:41:45 debian kernel: [63592.734252] RBP: ffffc15ec2feba70 R08: 0000000000000000 R09: ffff9f33c14e98c0
Feb 28 16:41:45 debian kernel: [63592.734254] R10: 0000000000000004 R11: 0000000000000000 R12: ffff9f37ee40b768
Feb 28 16:41:45 debian kernel: [63592.734255] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
Feb 28 16:41:45 debian kernel: [63592.734257] FS:  00007f8ca33f96c0(0000) GS:ffff9f3b1fc00000(0000) knlGS:0000000000000000
Feb 28 16:41:45 debian kernel: [63592.734261] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 28 16:41:45 debian kernel: [63592.734263] CR2: 00007f14227aa000 CR3: 00000001088cc002 CR4: 00000000003706e0
Feb 28 16:41:45 debian kernel: [63592.734266] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 28 16:41:45 debian kernel: [63592.734268] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Feb 28 16:41:45 debian kernel: [63592.734270] Call Trace:
Feb 28 16:41:45 debian kernel: [63592.734273]  <TASK>
Feb 28 16:41:45 debian kernel: [63592.734277]  _raw_spin_lock_irqsave+0x39/0x50
Feb 28 16:41:45 debian kernel: [63592.734283]  __wake_up_common_lock+0x67/0xd0
Feb 28 16:41:45 debian kernel: [63592.734294]  add_stream_packet+0x159/0x220 [xt_RTPENGINE]
Feb 28 16:41:45 debian kernel: [63592.734309]  proc_control_read_write+0x11a0/0x20b0 [xt_RTPENGINE]
Feb 28 16:41:45 debian kernel: [63592.734321]  ? put_cmsg+0x79/0x1a0
Feb 28 16:41:45 debian kernel: [63592.734329]  ? __slab_free+0xbf/0x2d0
Feb 28 16:41:45 debian kernel: [63592.734344]  proc_reg_write+0x56/0xa0
Feb 28 16:41:45 debian kernel: [63592.734351]  vfs_write+0xc7/0x3f0
Feb 28 16:41:45 debian kernel: [63592.734358]  ? __seccomp_filter+0x32a/0x4e0
Feb 28 16:41:45 debian kernel: [63592.734365]  ? __fget_light+0x9d/0x100
Feb 28 16:41:45 debian kernel: [63592.734372]  ksys_write+0xb7/0xf0
Feb 28 16:41:45 debian kernel: [63592.734379]  do_syscall_64+0x55/0xb0
Feb 28 16:41:45 debian kernel: [63592.734387]  ? exit_to_user_mode_prepare+0x40/0x1e0
Feb 28 16:41:45 debian kernel: [63592.734393]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 28 16:41:45 debian kernel: [63592.734397]  ? do_syscall_64+0x61/0xb0
Feb 28 16:41:45 debian kernel: [63592.734402]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 28 16:41:45 debian kernel: [63592.734405]  ? do_syscall_64+0x61/0xb0
Feb 28 16:41:45 debian kernel: [63592.734410]  ? do_syscall_64+0x61/0xb0
Feb 28 16:41:45 debian kernel: [63592.734414]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Feb 28 16:41:45 debian kernel: [63592.734420] RIP: 0033:0x7f8d2fffb37f
Feb 28 16:41:45 debian kernel: [63592.734424] Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 b9 d4 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 0c d5 f8 ff 48
Feb 28 16:41:45 debian kernel: [63592.734426] RSP: 002b:00007f8ca33f7840 EFLAGS: 00000297 ORIG_RAX: 0000000000000001
Feb 28 16:41:45 debian kernel: [63592.734430] RAX: ffffffffffffffda RBX: 00007f8ca33f7c00 RCX: 00007f8d2fffb37f
Feb 28 16:41:45 debian kernel: [63592.734432] RDX: 0000000000000078 RSI: 00007f8ca33f7870 RDI: 0000000000000004
Feb 28 16:41:45 debian kernel: [63592.734434] RBP: 00007f8ca33f7950 R08: 0000000000000001 R09: 0000000000000000
Feb 28 16:41:45 debian kernel: [63592.734436] R10: 00007f8ca33f7de0 R11: 0000000000000297 R12: 00007f8ca33f7900
Feb 28 16:41:45 debian kernel: [63592.734438] R13: 0000000000000050 R14: 000000000000001c R15: 00007f8ca33f787c
Feb 28 16:41:45 debian kernel: [63592.734443]  </TASK>
Feb 28 16:41:49 debian kernel: [63596.744994] watchdog: BUG: soft lockup - CPU#21 stuck for 26s! [poller:1319]
Feb 28 16:41:49 debian kernel: [63596.745335] Modules linked in: xt_RTPENGINE(OE) xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_set nft_compat nf_tables libcrc32c ip_set_hash_net ip_set nfnetlink sunrpc intel_rapl_msr intel_rapl_common binfmt_misc intel_uncore_frequency intel_uncore_frequency_common ipmi_ssif sb_edac nls_ascii x86_pkg_temp_thermal nls_cp437 intel_powerclamp vfat coretemp fat kvm_intel kvm irqbypass ghash_clmulni_intel sha512_ssse3 sha512_generic sha256_ssse3 sha1_ssse3 aesni_intel ledtrig_audio dell_smbios crypto_simd cryptd dell_wmi_descriptor mgag200 rfkill rapl drm_shmem_helper video iTCO_wdt ipmi_si drm_kms_helper intel_cstate mei_me intel_pmc_bxt iTCO_vendor_support ipmi_devintf sg mei i2c_algo_bit intel_uncore watchdog evdev pcspkr ipmi_msghandler dcdbas mxm_wmi acpi_power_meter button drm fuse configfs loop efi_pstore efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_mod sd_mod t10_pi sr_mod cdrom crc64_rocksoft crc64 crc_t10dif crct10dif_generic ahci
Feb 28 16:41:49 debian kernel: [63596.745387]  libahci ehci_pci libata ehci_hcd megaraid_sas crct10dif_pclmul tg3 crct10dif_common crc32_pclmul usbcore crc32c_intel scsi_mod libphy lpc_ich usb_common scsi_common wmi
Feb 28 16:41:49 debian kernel: [63596.748327] CPU: 21 PID: 1319 Comm: poller Tainted: G      D    OE      6.1.0-38-amd64 #1  Debian 6.1.147-1
Feb 28 16:41:49 debian kernel: [63596.748655] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 2.19.0 02/22/2024
Feb 28 16:41:49 debian kernel: [63596.748978] RIP: 0010:smp_call_function_many_cond+0xf4/0x330
Feb 28 16:41:49 debian kernel: [63596.749306] Code: 63 d0 e8 3f 33 3c 00 3b 05 c9 e2 ac 01 73 25 48 63 d0 48 8b 33 48 03 34 d5 60 9b e1 9f 8b 56 08 83 e2 01 74 0a f3 90 8b 4e 08 <83> e1 01 75 f6 83 c0 01 eb c1 48 83 c4 38 5b 5d 41 5c 41 5d 41 5e
Feb 28 16:41:49 debian kernel: [63596.749980] RSP: 0018:ffffc15ec2f93c60 EFLAGS: 00000202
Feb 28 16:41:49 debian kernel: [63596.750328] RAX: 0000000000000018 RBX: ffff9f3b1fb72c80 RCX: 0000000000000011
Feb 28 16:41:49 debian kernel: [63596.750677] RDX: 0000000000000001 RSI: ffff9f3b1fc39b40 RDI: ffff9f33c0f77260
Feb 28 16:41:49 debian kernel: [63596.751026] RBP: ffff9f3b1fb71440 R08: 0000000000000018 R09: ffff9f33c0f77aa0
Feb 28 16:41:49 debian kernel: [63596.751378] R10: 00007f8cd02fdfff R11: 0000000000000000 R12: ffff9f3b1fb72c80
Feb 28 16:41:49 debian kernel: [63596.751729] R13: 0000000000000001 R14: 0000000000000015 R15: 0000000000000001
Feb 28 16:41:49 debian kernel: [63596.752082] FS:  00007f8d1cde66c0(0000) GS:ffff9f3b1fb40000(0000) knlGS:0000000000000000
Feb 28 16:41:49 debian kernel: [63596.752436] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 28 16:41:49 debian kernel: [63596.752796] CR2: 00005558b1049000 CR3: 00000001088cc001 CR4: 00000000003706e0
Feb 28 16:41:49 debian kernel: [63596.753157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 28 16:41:49 debian kernel: [63596.753520] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Feb 28 16:41:49 debian kernel: [63596.753883] Call Trace:
Feb 28 16:41:49 debian kernel: [63596.754246]  <TASK>
Feb 28 16:41:49 debian kernel: [63596.754608]  ? leave_mm+0x40/0x40
Feb 28 16:41:49 debian kernel: [63596.754973]  on_each_cpu_cond_mask+0x20/0x40
Feb 28 16:41:49 debian kernel: [63596.755338]  flush_tlb_mm_range+0xf0/0x170
Feb 28 16:41:49 debian kernel: [63596.755702]  tlb_finish_mmu+0xe6/0x170
Feb 28 16:41:49 debian kernel: [63596.756067]  zap_page_range_single+0x123/0x1a0
Feb 28 16:41:49 debian kernel: [63596.756434]  do_madvise+0xa78/0xf00
Feb 28 16:41:49 debian kernel: [63596.756802]  __x64_sys_madvise+0x28/0x30
Feb 28 16:41:49 debian kernel: [63596.757169]  do_syscall_64+0x55/0xb0
Feb 28 16:41:49 debian kernel: [63596.757538]  ? do_syscall_64+0x61/0xb0
Feb 28 16:41:49 debian kernel: [63596.757902]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 28 16:41:49 debian kernel: [63596.758257]  ? do_syscall_64+0x61/0xb0
Feb 28 16:41:49 debian kernel: [63596.758600]  ? do_syscall_64+0x61/0xb0
Feb 28 16:41:49 debian kernel: [63596.758937]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Feb 28 16:41:49 debian kernel: [63596.759273] RIP: 0033:0x7f8d30004af7
Feb 28 16:41:49 debian kernel: [63596.759606] Code: ff ff ff ff c3 66 0f 1f 44 00 00 48 8b 15 01 13 0d 00 f7 d8 64 89 02 b8 ff ff ff ff eb bc 0f 1f 44 00 00 b8 1c 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d9 12 0d 00 f7 d8 64 89 01 48
Feb 28 16:41:49 debian kernel: [63596.760307] RSP: 002b:00007f8d1cde4ac8 EFLAGS: 00000246 ORIG_RAX: 000000000000001c
Feb 28 16:41:49 debian kernel: [63596.760672] RAX: ffffffffffffffda RBX: 0000000000027830 RCX: 00007f8d30004af7
Feb 28 16:41:49 debian kernel: [63596.761032] RDX: 0000000000000004 RSI: 0000000000007000 RDI: 00007f8cd02f7000
Feb 28 16:41:49 debian kernel: [63596.761389] RBP: 00007f8cd0000030 R08: 0000000000000007 R09: 0000000000000000
Feb 28 16:41:49 debian kernel: [63596.761739] R10: 0000000000000250 R11: 0000000000000246 R12: 00007f8cd0000000
Feb 28 16:41:49 debian kernel: [63596.762082] R13: 00000000002f7000 R14: 0000000000007000 R15: 00007f8cd02d67d0
Feb 28 16:41:49 debian kernel: [63596.762418]  </TASK>

[rfuchs]

Could this be a hardware issue?

Depends. With faulty hardware you'd expect very random crashes. If your crash dumps always indicate the same location, then it's likely not the hardware.

[rfuchs]

Try with 8aad5e2. Bit of a shot in the dark, but it might just help.

Edit: No, doesn't look like it helps.

[Young-LLL-6130]

Try with 8aad5e2. Bit of a shot in the dark, but it might just help.

Edit: No, doesn't look like it helps.

Well, we initially suspected a hardware issue and replaced the server, but the same kernel error persists. Looks like epoll_ctl is triggering the issue.

[rfuchs]

Somewhat confident that d1efc8d should be the fix. Try it if you can and let me know.

[Young-LLL-6130]

@rfuchs Okay, I have started the testing. I will provide feedback on the test results later.