scx-lavd 1.0.19 (and 1.0.18) crashed with `cannot move migration disabled`

[kakra]

This issue has not been present in 1.0.16.

I've seen it previously in 1.0.18 but wanted to wait for the next kernel and scx version before I report it. Now, here we are.

This usually happens easily with Chrome while playing Youtube videos. The scheduler crashes and will eventually be restarted by systemd until the restart counter expires:

# uname -a
Linux jupiter 6.18.3-gentoo #2 SMP PREEMPT Sun Jan  4 15:13:30 CET 2026 x86_64 12th Gen Intel(R) Core(TM) i7-12700K GenuineIntel GNU/Linux

# lscpu
Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             46 bits physical, 48 bits virtual
  Byte Order:                Little Endian
CPU(s):                      20
  On-line CPU(s) list:       0-19
Vendor ID:                   GenuineIntel
  Model name:                12th Gen Intel(R) Core(TM) i7-12700K
    CPU family:              6
    Model:                   151
    Thread(s) per core:      2
    Core(s) per socket:      12
    Socket(s):               1
    Stepping:                2
    CPU(s) scaling MHz:      19%
    CPU max MHz:             5000.0000
    CPU min MHz:             800.0000
    BogoMIPS:                7222.00
    Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4
                             _2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni dtherm ida arat pln pts hwp
                             hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ibt flush_l1d arch_capabilities
Virtualization features:
  Virtualization:            VT-x
Caches (sum of all):
  L1d:                       512 KiB (12 instances)
  L1i:                       512 KiB (12 instances)
  L2:                        12 MiB (9 instances)
  L3:                        25 MiB (1 instance)
Vulnerabilities:
  Gather data sampling:      Not affected
  Ghostwrite:                Not affected
  Indirect target selection: Not affected
  Itlb multihit:             Not affected
  L1tf:                      Not affected
  Mds:                       Not affected
  Meltdown:                  Not affected
  Mmio stale data:           Not affected
  Old microcode:             Not affected
  Reg file data sampling:    Vulnerable
  Retbleed:                  Not affected
  Spec rstack overflow:      Not affected
  Spec store bypass:         Vulnerable
  Spectre v1:                Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
  Spectre v2:                Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Vulnerable; BHI: Vulnerable
  Srbds:                     Not affected
  Tsa:                       Not affected
  Tsx async abort:           Not affected
  Vmscape:                   Vulnerable

I often see NOHZ tick-stop error: local softirq work is pending when stopping scx but this one also captures the crash itself:

# dmesg upgrading from 1.0.16 to 1.0.19
[209999.331435] NOHZ tick-stop error: local softirq work is pending, handler #200!!!
[209999.333611] NOHZ tick-stop error: local softirq work is pending, handler #280!!!
[209999.336930] NOHZ tick-stop error: local softirq work is pending, handler #280!!!
[209999.336933] NOHZ tick-stop error: local softirq work is pending, handler #280!!!
[209999.340268] NOHZ tick-stop error: local softirq work is pending, handler #280!!!
[209999.340270] NOHZ tick-stop error: local softirq work is pending, handler #280!!!
[209999.340489] NOHZ tick-stop error: local softirq work is pending, handler #200!!!
[209999.351920] NOHZ tick-stop error: local softirq work is pending, handler #80!!!
[209999.353610] NOHZ tick-stop error: local softirq work is pending, handler #80!!!
[209999.353769] NOHZ tick-stop error: local softirq work is pending, handler #80!!!
[209999.483712] sched_ext: BPF scheduler "lavd_1.0.16_x86_64_unknown_linux_gnu" disabled (unregistered from user space)
[210002.820652] sched_ext: BPF scheduler "lavd_1.0.20_x86_64_unknown_linux_gnu" enabled
[210002.820695] sched_ext: scx_bpf_cpu_rq() is deprecated; use scx_bpf_locked_rq() when holding rq lock or scx_bpf_cpu_curr() to read remote curr safely.
[210051.375101] sched_ext: BPF scheduler "lavd_1.0.20_x86_64_unknown_linux_gnu" disabled (runtime error)
[210051.375111] sched_ext: lavd_1.0.20_x86_64_unknown_linux_gnu: SCX_DSQ_LOCAL[_ON] cannot move migration disabled chrome[561481] from CPU 18 to 10
[210051.375115]    task_can_run_on_remote_rq+0xd0/0x130
[210051.375126]    dispatch_to_local_dsq+0x61/0x200
[210051.375128]    process_ddsp_deferred_locals+0x49/0xc0
[210051.375129]    deferred_irq_workfn+0x23/0x30
[210051.375130]    irq_work_run_list+0x49/0x80
[210051.375132]    irq_work_run+0x18/0x40
[210051.375133]    __sysvec_irq_work+0x1c/0xb0
[210051.375135]    sysvec_irq_work+0x6c/0x90
[210051.375137]    asm_sysvec_irq_work+0x1a/0x20
[210051.375138]    finish_task_switch.isra.0+0x90/0x260
[210051.375140]    __schedule+0x3d5/0x1020
[210051.375141]    schedule+0x27/0xd0
[210051.375143]    schedule_hrtimeout_range_clock+0xdc/0xe0
[210051.375144]    do_epoll_wait+0x443/0x480
[210051.375146]    __x64_sys_epoll_wait+0x51/0xe0
[210051.375148]    do_syscall_64+0x52/0xfd0
[210051.375149]    entry_SYSCALL_64_after_hwframe+0x4b/0x53
[210055.008059] sched_ext: scx_bpf_cpu_rq() is deprecated; use scx_bpf_locked_rq() when holding rq lock or scx_bpf_cpu_curr() to read remote curr safely.
[210055.008379] sched_ext: BPF scheduler "lavd_1.0.20_x86_64_unknown_linux_gnu" enabled

Take note that the scheduler identifies as 1.0.20 although I've installed 1.0.19:

# equery l scx
 * Searching for scx ...
[IP-] [  ] sys-kernel/scx-1.0.19:0

This already happened with the previous version which showed 1.0.19 despite 1.0.18 being installed.

I'm using the new scx_loader.service for both 1.0.16 and 1.0.19:

# systemctl status scx_loader.service
● scx_loader.service - DBUS on-demand loader of sched-ext schedulers
     Loaded: loaded (/usr/lib/systemd/system/scx_loader.service; enabled; preset: disabled)
     Active: active (running) since Wed 2026-01-07 01:40:24 CET; 1min 36s ago
 Invocation: bb06dd9a12aa494a90b1fc6bb999aeb6
   Main PID: 2183383 (scx_loader)
      Tasks: 26 (limit: 38072)
        CPU: 2.077s
     CGroup: /system.slice/scx_loader.service
             ├─2183383 /usr/bin/scx_loader
             └─2188865 scx_lavd

Jan 07 01:41:16 jupiter scx_loader[2188865]: }
Jan 07 01:41:16 jupiter scx_loader[2188865]: 2026-01-07T00:41:16.985319Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:541: capacity bound:  778 (4.0428185%)
Jan 07 01:41:16 jupiter scx_loader[2188865]:   primary CPUs:  [16]
Jan 07 01:41:16 jupiter scx_loader[2188865]:   overflow CPUs: [17, 18, 19, 0, 1, 2, 3, 4, 5, 6, 7, 12, 13, 14, 15, 8, 9, 10, 11]
Jan 07 01:41:16 jupiter scx_loader[2188865]: 2026-01-07T00:41:16.985329Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:541: capacity bound:  19244 (100%)
Jan 07 01:41:16 jupiter scx_loader[2188865]:   primary CPUs:  [8]
Jan 07 01:41:16 jupiter scx_loader[2188865]:   overflow CPUs: [10, 0, 2, 4, 6, 12, 14, 16, 17, 18, 19, 9, 11, 1, 3, 5, 7, 13, 15]
Jan 07 01:41:18 jupiter scx_loader[2188865]: 2026-01-07T00:41:18.035071Z  WARN ThreadId(01) scx_utils::libbpf_logger: rust/scx_utils/src/libbpf_logger.rs:12: libbpf: map 'lavd_ops': BPF map skeleton link is uninitialized
Jan 07 01:41:18 jupiter scx_loader[2188865]: 2026-01-07T00:41:18.121319Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:1010: scx_lavd scheduler is initialized (build ID: 1.0.20 x86_64-unknown-linux-gnu)
Jan 07 01:41:18 jupiter scx_loader[2188865]: 2026-01-07T00:41:18.121337Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:1014: scx_lavd scheduler starts running.

[kakra]

Maybe related: #3180

[kakra]

With a later kernel:

[   13.122463] sched_ext: scx_bpf_cpu_rq() is deprecated; use scx_bpf_locked_rq() when holding rq lock or scx_bpf_cpu_curr() to read remote curr safely.
[   13.122487] sched_ext: BPF scheduler "lavd_1.0.20_x86_64_unknown_linux_gnu" enabled
[   14.894717] ------------[ cut here ]------------
[   14.894719] WARNING: CPU: 2 PID: 2556 at kernel/sched/ext.c:3203 scx_cgroup_move_task+0x9d/0xc0
[   14.894725] Modules linked in: intel_tcc_cooling coretemp kvm_intel snd_hda_codec_intelhdmi kvm snd_hda_codec_hdmi snd_hda_codec_alc662 snd_hda_codec_realtek_lib snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg snd_hda_codec snd_hda_core snd_usb_audio snd_hwdep tpm_crb tpm_tis snd_usbmidi_lib uvcvideo tpm_tis_core snd_rawmidi videobuf2_vmalloc snd_seq_device uvc snd_pcm videobuf2_memops videobuf2_v4l2 snd_timer videodev btusb btintel bluetooth input_leds snd videobuf2_common mc led_class soundcore rfkill crc16 nfsd nfs_acl lockd auth_rpcgss fuse grace nfnetlink tpm libaescfb sunrpc
[   14.894754] CPU: 2 UID: 0 PID: 2556 Comm: systemd-nspawn Not tainted 6.18.5-gentoo #1 PREEMPT(lazy)
[   14.894756] Hardware name: Wortmann_AG 1009698;2120150/B360M-D3P-WG-CF, BIOS F13 WG 07/04/2019
[   14.894758] Sched_ext: lavd_1.0.20_x86_64_unknown_linux_gnu (enabled+all), task: runnable_at=+0ms
[   14.894759] RIP: 0010:scx_cgroup_move_task+0x9d/0xc0
[   14.894762] Code: 24 ff d1 65 48 8b 05 32 5d bd 02 48 8b 3c 24 48 c7 80 00 03 00 00 00 00 00 00 48 c7 87 58 03 00 00 00 00 00 00 48 83 c4 08 c3 <0f> 0b 48 c7 87 58 03 00 00 00 00 00 00 c3 c3 48 c7 c2 00 d0 5d af
[   14.894763] RSP: 0018:ffff9d5782077c40 EFLAGS: 00010046
[   14.894765] RAX: ffff97f2a748c400 RBX: ffff97fa1e8aa8c0 RCX: ffff97f2abd62180
[   14.894766] RDX: 00000000e587f1ff RSI: ffff97f29cad2000 RDI: ffff97f2abd62100
[   14.894768] RBP: ffff97f2abd62100 R08: ffff97f28bef7480 R09: ffff97f2abd62100
[   14.894769] R10: ffff97f2abd62100 R11: 0000000000000000 R12: ffff97f2abd62100
[   14.894770] R13: 0000000000000001 R14: 0000000000000000 R15: ffff97f2abd62100
[   14.894771] FS:  000056236fc9c440(0000) GS:ffff97fa6e966000(0000) knlGS:0000000000000000
[   14.894772] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   14.894774] CR2: 000055c56db5d000 CR3: 0000000131471001 CR4: 00000000003726f0
[   14.894775] Call Trace:
[   14.894777]  <TASK>
[   14.894777]  sched_move_task+0x117/0x270
[   14.894784]  cpu_cgroup_attach+0x24/0x40
[   14.894786]  cgroup_migrate_execute+0x2c8/0x3f0
[   14.894789]  ? cgroup_migrate_prepare_dst+0x46/0x210
[   14.894791]  cgroup_update_dfl_csses+0x1d1/0x2a0
[   14.894794]  cgroup_subtree_control_write+0x21f/0x430
[   14.894796]  kernfs_fop_write_iter+0x12b/0x1e0
[   14.894800]  vfs_write+0x1fb/0x420
[   14.894804]  ksys_write+0x57/0xc0
[   14.894806]  do_syscall_64+0x4e/0x11c0
[   14.894809]  ? irq_exit_rcu+0x41/0xb0
[   14.894812]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
[   14.894814] RIP: 0033:0x56236f6af267
[   14.894815] Code: 89 fa 4c 89 df 48 8b 74 24 20 e8 a4 b4 00 00 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 16 5b c3 0f 1f 40 00 48 8b 44 24 10 0f 05 <5b> c3 0f 1f 80 00 00 00 00 83 e2 39 83 fa 08 75 e2 e8 23 ff ff ff
[   14.894817] RSP: 002b:00007ffe2983e480 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[   14.894818] RAX: ffffffffffffffda RBX: 000056236fc9c440 RCX: 000056236f6af267
[   14.894819] RDX: 0000000000000008 RSI: 0000562384e0aa80 RDI: 0000000000000012
[   14.894820] RBP: 0000562384e072a0 R08: 0000000000000000 R09: 0000000000000000
[   14.894821] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000008
[   14.894822] R13: 0000562384e0aa80 R14: 0000000000000008 R15: 000056236f7efee0
[   14.894824]  </TASK>
[   14.894825] ---[ end trace 0000000000000000 ]---

[kakra]

Still fails with latest git version d5712c0:

2026-01-18T02:44:21.424256Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:301: Autopilot mode is enabled.
2026-01-18T02:44:21.424367Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:341: Energy model won't be used for CPU preference order.
2026-01-18T02:44:21.424371Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:354: Pinned task slice mode is enabled (5000 us). Pinned tasks will use per-CPU DSQs.
2026-01-18T02:44:21.424373Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:977: Opts {
    verbose: 0,
    autopilot: true,
    autopower: false,
    performance: false,
    powersave: false,
    balanced: false,
    slice_max_us: 5000,
    slice_min_us: 500,
    mig_delta_pct: 0,
    pinned_slice_us: Some(
        5000,
    ),
    preempt_shift: 6,
    cpu_pref_order: "",
    no_use_em: true,
    no_futex_boost: false,
    no_preemption: false,
    no_wake_sync: false,
    no_slice_boost: false,
    per_cpu_dsq: false,
    enable_cpu_bw: false,
    no_core_compaction: false,
    no_freq_scaling: false,
    stats: None,
    monitor: None,
    monitor_sched_samples: None,
    log_level: "info",
    version: false,
    run_id: None,
    help_stats: false,
    libbpf: LibbpfOpts {
        relaxed_maps: None,
        pin_root_path: None,
        kconfig: None,
        btf_custom_path: None,
        bpf_token_path: None,
    },
    topology: None,
}
2026-01-18T02:44:21.460118Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:541: capacity bound:  778 (4.0428185%)
  primary CPUs:  [16]
  overflow CPUs: [17, 18, 19, 0, 1, 2, 3, 4, 5, 6, 7, 12, 13, 14, 15, 8, 9, 10, 11]
2026-01-18T02:44:21.460130Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:541: capacity bound:  19244 (100%)
  primary CPUs:  [8]
  overflow CPUs: [10, 0, 2, 4, 6, 12, 14, 16, 17, 18, 19, 9, 11, 1, 3, 5, 7, 13, 15]
2026-01-18T02:44:22.667501Z  WARN ThreadId(01) scx_utils::libbpf_logger: rust/scx_utils/src/libbpf_logger.rs:12: libbpf: map 'lavd_ops': BPF map skeleton link is uninitialized

2026-01-18T02:44:22.768071Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:1010: scx_lavd scheduler is initialized (build ID: 1.0.20-gd5712c02 x86_64-unknown-linux-gnu)
2026-01-18T02:44:22.768086Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:1014: scx_lavd scheduler starts running.

DEBUG DUMP
================================================================================

kwin_wayland[4429] triggered exit kind 1024:
  runtime error (SCX_DSQ_LOCAL[_ON] cannot move migration disabled Compositor[1374911] from CPU 0 to 8)

Backtrace:
  task_can_run_on_remote_rq+0xd0/0x130
  dispatch_to_local_dsq+0x61/0x200
  process_ddsp_deferred_locals+0x49/0xc0
  deferred_irq_workfn+0x23/0x30
  irq_work_run_list+0x49/0x80
  irq_work_run+0x18/0x40
  __sysvec_irq_work+0x1c/0xb0
  sysvec_irq_work+0x6c/0x90
  asm_sysvec_irq_work+0x1a/0x20
  finish_task_switch.isra.0+0x90/0x260
  __schedule+0x3d5/0x1020
  schedule+0x27/0xd0
  schedule_hrtimeout_range_clock+0xdc/0xe0
  poll_schedule_timeout.constprop.0+0x37/0x80
  do_sys_poll+0x3ad/0x540
  __x64_sys_ppoll+0xaa/0x150
  do_syscall_64+0x52/0xfd0
  entry_SYSCALL_64_after_hwframe+0x4b/0x53

CPU states
----------

CPU 0   : nr_run=1 flags=0x9 cpu_rel=1 ops_qseq=56225538 pnt_seq=231627263
          curr=kwin_wayland[4429] class=rt_sched_class

  R Compositor[1374911] +0ms
      scx_state/flags=3/0x1 dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=0x8000000000000001
      dsq_vtime=44518051 slice=500000 weight=596
      cpus=00001 no_mig=1

CPU 13  : nr_run=1 flags=0x1 cpu_rel=0 ops_qseq=18633603 pnt_seq=52247978
          curr=borg[1934828] class=ext_sched_class

 *R borg[1934828] +0ms
      scx_state/flags=3/0x5 dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=(n/a)
      dsq_vtime=44493480 slice=4957423 weight=4
      cpus=fffff no_mig=0

Event counters
--------------
              SCX_EV_SELECT_CPU_FALLBACK:                0
       SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE:                0
               SCX_EV_DISPATCH_KEEP_LAST:                0
                 SCX_EV_ENQ_SKIP_EXITING:                0
      SCX_EV_ENQ_SKIP_MIGRATION_DISABLED:                0
                 SCX_EV_REFILL_SLICE_DFL:               41
                  SCX_EV_BYPASS_DURATION:         95617987
                  SCX_EV_BYPASS_DISPATCH:               41
                  SCX_EV_BYPASS_ACTIVATE:                1

================================================================================

2026-01-18T02:44:24.768260Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:909: Unregister scx_lavd scheduler
Error: EXIT: runtime error (SCX_DSQ_LOCAL[_ON] cannot move migration disabled Compositor[1374911] from CPU 0 to 8)

[WaxArsatia]

Just trying scx-lavd and installing CachyOS today, checking my dmesg, kind of same here, with mitigations=off kernel parameter.

# uname -a
Linux cachyos 6.18.7-2-cachyos #1 SMP PREEMPT_DYNAMIC Sat, 24 Jan 2026 21:21:21 +0000 x86_64 GNU/Linux

# scx_lavd --version
scx_lavd 1.0.20-g85f140fd x86_64-unknown-linux-gnu

# dmesg
[   10.894007] sched_ext: BPF scheduler "lavd_1.0.20_g85f140fd_x86_64_unknown_linux_gnu" enabled
[   10.895979] sched_ext: scx_bpf_cpu_rq() is deprecated; use scx_bpf_locked_rq() when holding rq lock or scx_bpf_cpu_curr() to read remote curr safely.
[   17.336349] ------------[ cut here ]------------
[   17.336353] WARNING: CPU: 4 PID: 1038 at kernel/sched/ext.c:3222 scx_cgroup_move_task+0x96/0xa0
[   17.336362] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq rfcomm snd_seq_device cmac algif_hash algif_skcipher af_alg snd_hda_codec_intelhdmi bnep snd_hda_codec_alc269 snd_hda_scodec_component snd_hda_codec_realtek_lib snd_hda_codec_generic snd_hda_intel snd_sof_pci_intel_cnl snd_sof_intel_hda_generic soundwire_intel soundwire_cadence snd_sof_intel_hda_sdw_bpt snd_sof_intel_hda_common snd_soc_hdac_hda snd_sof_intel_hda snd_hda_codec_hdmi snd_sof_intel_hda_mlink snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_acpi_intel_match snd_soc_acpi_intel_sdca_quirks soundwire_generic_allocation intel_uncore_frequency snd_soc_acpi intel_uncore_frequency_common soundwire_bus snd_soc_sdca crc8 intel_tcc_cooling snd_soc_avs x86_pkg_temp_thermal snd_soc_hda_codec intel_powerclamp snd_hda_ext_core coretemp snd_hda_codec snd_hda_core snd_intel_dspcfg kvm_intel iwlmvm snd_intel_sdw_acpi snd_hwdep processor_thermal_device_pci_legacy kvm snd_soc_core processor_thermal_device btusb mac80211 ac97_bus
[   17.336408]  processor_thermal_power_floor uvcvideo btmtk processor_thermal_wt_hint snd_pcm_dmaengine r8169 uvc processor_thermal_wt_req irqbypass snd_compress videobuf2_vmalloc btbcm polyval_clmulni realtek processor_thermal_rfim videobuf2_memops ghash_clmulni_intel snd_pcm btintel ptp ee1004 asus_nb_wmi processor_thermal_mbox videobuf2_v4l2 aesni_intel pps_core mdio_devres platform_temperature_control mousedev intel_rapl_msr libarc4 processor_thermal_rapl rapl mei_pxp asus_armoury mei_hdcp snd_timer i2c_i801 btrtl intel_cstate videobuf2_common intel_rapl_common vfat libphy spi_nor iwlwifi ntfs3 fat i2c_smbus videodev mei_me processor_thermal_soc_slider snd hid_multitouch intel_uncore firmware_attributes_class psmouse pcspkr wmi_bmof mdio_bus mtd bluetooth soundcore cfg80211 i2c_mux mc mei int340x_thermal_zone i2c_hid_acpi intel_soc_dts_iosf intel_pmc_core intel_pch_thermal i2c_hid intel_oc_wdt intel_pmc_ssram_telemetry pmt_telemetry pmt_discovery pinctrl_cannonlake pmt_class int3400_thermal joydev intel_vsec
[   17.336466]  acpi_thermal_rel acpi_pad asus_wireless mac_hid dm_mod i2c_dev ntsync pkcs8_key_parser crypto_user nfnetlink zram 842_compress 842_decompress lz4hc_compress lz4_compress nvidia_drm(POE) nvidia_uvm(POE) nvidia_modeset(POE) xfs nvidia(POE) i915 nvme rtsx_pci_sdmmc drm_buddy nvme_core mmc_core intel_gtt i2c_algo_bit nvme_keyring hid_asus nvme_auth spi_intel_pci intel_lpss_pci drm_display_helper asus_wmi rtsx_pci hkdf intel_lpss spi_intel sparse_keymap cec idma64 platform_profile rfkill serio_raw mxm_wmi drm_ttm_helper ttm video wmi
[   17.336505] CPU: 4 UID: 1000 PID: 1038 Comm: systemd Tainted: P           OE       6.18.7-2-cachyos #1 PREEMPT(full)  adc06fafc8215c9c47ab51672efc583f40cbf3be
[   17.336511] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[   17.336513] Hardware name: ASUSTeK COMPUTER INC. Strix 17 GL703GE/GL703GE, BIOS GL703GE.321 01/15/2021
[   17.336515] Sched_ext: lavd_1.0.20_g85f140fd_x86_64_unknown_linux_gnu (enabled+all), task: runnable_at=-1ms
[   17.336517] RIP: 0010:scx_cgroup_move_task+0x96/0xa0
[   17.336521] Code: 89 fb 2e 2e 2e 41 ff d3 48 89 df 49 c7 86 00 03 00 00 00 00 00 00 5b 41 5e 48 c7 87 60 03 00 00 00 00 00 00 c3 cc cc cc cc cc <0f> 0b eb eb cc cc cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90
[   17.336524] RSP: 0018:ffffd3c4a008fb30 EFLAGS: 00010046
[   17.336526] RAX: ffff8cd90adf7c00 RBX: ffff8cdc6dc31e40 RCX: ffff8cd911169f00
[   17.336528] RDX: ffff8cd9211e3e00 RSI: ffff8cd9211e3e00 RDI: ffff8cd917f4d280
[   17.336530] RBP: 0000000000000000 R08: 0000000000000002 R09: 0000000000000002
[   17.336532] R10: 0000000000000014 R11: 0000000000000000 R12: ffff8cd900ce3700
[   17.336533] R13: 0000000000000000 R14: ffff8cd917f4d280 R15: ffff8cd923787c00
[   17.336535] FS:  00007fb295d1afc0(0000) GS:ffff8cdce6c07000(0000) knlGS:0000000000000000
[   17.336537] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   17.336539] CR2: 00005557f9439b40 CR3: 0000000116296003 CR4: 00000000003726f0
[   17.336541] Call Trace:
[   17.336543]  <TASK>
[   17.336545]  sched_move_task+0x272/0x310
[   17.336553]  cpu_cgroup_attach+0x4a/0x80
[   17.336557]  cgroup_migrate_execute+0x2b8/0x3c0
[   17.336562]  cgroup_apply_control+0x524/0x570
[   17.336569]  cgroup_subtree_control_write+0x4a6/0x570
[   17.336574]  kernfs_fop_write_iter.llvm.16430091695160210628+0x10d/0x190
[   17.336579]  __x64_sys_write+0x25c/0x430
[   17.336585]  do_syscall_64+0x6b/0x210
[   17.336589]  ? __x64_sys_write+0x3be/0x430
[   17.336594]  ? do_syscall_64+0xa4/0x210
[   17.336597]  ? do_syscall_64+0xa4/0x210
[   17.336599]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[   17.336602] RIP: 0033:0x7fb2950a432a
[   17.336632] Code: 8b 05 e2 29 19 00 64 89 10 48 c7 c2 ff ff ff ff 48 83 c4 18 48 89 d0 c3 0f 1f 84 00 00 00 00 00 49 89 ca 48 8b 44 24 20 0f 05 <48> 89 c2 48 3d 00 f0 ff ff 77 0b 48 89 d0 48 83 c4 18 c3 0f 1f 00
[   17.336634] RSP: 002b:00007ffffd1c5210 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[   17.336637] RAX: ffffffffffffffda RBX: 00005557f9395650 RCX: 00007fb2950a432a
[   17.336639] RDX: 0000000000000004 RSI: 00005557f9496520 RDI: 0000000000000014
[   17.336640] RBP: 0000000000000004 R08: 0000000000000000 R09: 0000000000000000
[   17.336641] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000004
[   17.336643] R13: 00005557f9496520 R14: 00005557f9496520 R15: 00007ffffd1c5370
[   17.336647]  </TASK>
[   17.336648] ---[ end trace 0000000000000000 ]---

[kakra]

Still an issue:

[853288.010344] sched_ext: BPF scheduler "lavd_1.0.21_x86_64_unknown_linux_gnu" disabled (runtime error)
[853288.010355] sched_ext: lavd_1.0.21_x86_64_unknown_linux_gnu: SCX_DSQ_LOCAL[_ON] cannot move migration disabled VideoFrameCompo[7036] from CPU 12 to 10
[853288.010359]    task_can_run_on_remote_rq+0xd0/0x130
[853288.010376]    dispatch_to_local_dsq+0x61/0x200
[853288.010385]    process_ddsp_deferred_locals+0x49/0xc0
[853288.010393]    deferred_irq_workfn+0x23/0x30
[853288.010401]    irq_work_run_list+0x49/0x80
[853288.010408]    irq_work_run+0x18/0x40
[853288.010414]    __sysvec_irq_work+0x1c/0xb0
[853288.010421]    sysvec_irq_work+0x6c/0x90
[853288.010429]    asm_sysvec_irq_work+0x1a/0x20
[853288.010435]    finish_task_switch.isra.0+0x90/0x260
[853288.010440]    __schedule+0x3d5/0x1020
[853288.010444]    schedule+0x27/0xd0
[853288.010447]    schedule_hrtimeout_range_clock+0x9d/0xe0
[853288.010454]    poll_schedule_timeout.constprop.0+0x37/0x80
[853288.010463]    do_sys_poll+0x3ad/0x540
[853288.010471]    __x64_sys_ppoll+0xaa/0x150
[853288.010478]    do_syscall_64+0x52/0xfd0
[853288.010483]    entry_SYSCALL_64_after_hwframe+0x4b/0x53
[853291.968862] sched_ext: BPF scheduler "lavd_1.0.21_x86_64_unknown_linux_gnu" enabled
[853349.791312] sched_ext: BPF scheduler "lavd_1.0.21_x86_64_unknown_linux_gnu" disabled (runtime error)
[853349.791317] sched_ext: lavd_1.0.21_x86_64_unknown_linux_gnu: SCX_DSQ_LOCAL[_ON] cannot move migration disabled sh[29761] from CPU 4 to 0
[853349.791318]    task_can_run_on_remote_rq+0xd0/0x130
[853349.791324]    dispatch_to_local_dsq+0x61/0x200
[853349.791326]    process_ddsp_deferred_locals+0x49/0xc0
[853349.791328]    deferred_irq_workfn+0x23/0x30
[853349.791330]    irq_work_run_list+0x49/0x80
[853349.791332]    irq_work_run+0x18/0x40
[853349.791333]    __sysvec_irq_work+0x1c/0xb0
[853349.791335]    sysvec_irq_work+0x6c/0x90
[853349.791338]    asm_sysvec_irq_work+0x1a/0x20
[853349.791340]    finish_task_switch.isra.0+0x90/0x260
[853349.791342]    __schedule+0x3d5/0x1020
[853349.791343]    schedule+0x27/0xd0
[853349.791344]    irq_thread+0xa5/0x220
[853349.791346]    kthread+0xec/0x1e0
[853349.791347]    ret_from_fork+0x11f/0x150
[853349.791349]    ret_from_fork_asm+0x11/0x20
[853353.825875] sched_ext: BPF scheduler "lavd_1.0.21_x86_64_unknown_linux_gnu" enabled

I wonder why it works for others when looking at some of the performance regressions in the issue reports: Seems to not crash there.

Are there any special other kernel config options needed? Is it because of threaded irqs? Or because I'm running irqbalance?

[Hill-98]

same problem.

kernel version: 6.19.6-1-cachyos-server

scx lavd version: 1.1.0-gc505008f (archlinux official package)

kernel log:

[  146.560332] ops->cpu_acquire/release() are deprecated, use sched_switch TP instead
[  146.592626] sched_ext: BPF scheduler "lavd_1.1.0_gc505008f_x86_64_unknown_linux_gnu" enabled
[  184.379557] ------------[ cut here ]------------
[  184.379564] !p->scx.cgrp_moving_from
[  184.379566] WARNING: kernel/sched/ext.c:3295 at scx_cgroup_move_task+0xa8/0xd0, CPU#0: crun/7600
[  184.379580] Modules linked in: tls tcp_diag udp_diag inet_diag overlay vhost_net vhost vhost_iotlb tap tun wireguard libcurve25519 ip6_udp_tunnel udp_tunnel bridge stp llc nft_masq nft_socket nf_socket_ipv4 nf_socket_ipv6 nft_limit nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_redir nft_tproxy nf_tproxy_ipv6 nf_tproxy_ipv4 nf_log_syslog nft_log nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables it87(OE) hwmon_vid vfat fat snd_hda_codec_intelhdmi snd_hda_codec_alc662 snd_hda_codec_realtek_lib snd_hda_codec_generic snd_hda_intel snd_sof_pci_intel_tgl snd_sof_pci_intel_cnl snd_sof_intel_hda_generic soundwire_intel snd_sof_intel_hda_sdw_bpt snd_sof_intel_hda_common snd_soc_hdac_hda snd_sof_intel_hda_mlink snd_sof_intel_hda snd_hda_codec_hdmi soundwire_cadence snd_sof_pci snd_sof_xtensa_dsp snd_sof intel_rapl_msr snd_sof_utils snd_soc_acpi_intel_match intel_rapl_common snd_soc_acpi_intel_sdca_quirks soundwire_generic_allocation snd_soc_sdw_utils snd_soc_acpi soundwire_bus snd_soc_sdca
[  184.379656]  crc8 x86_pkg_temp_thermal snd_soc_avs intel_powerclamp snd_soc_hda_codec snd_hda_ext_core coretemp snd_hda_codec snd_hda_core snd_intel_dspcfg kvm_intel iwlmvm snd_intel_sdw_acpi snd_hwdep snd_soc_core kvm mac80211 snd_compress btusb irqbypass btmtk ac97_bus btrtl libarc4 iTCO_wdt ghash_clmulni_intel intel_pmc_bxt btbcm aesni_intel snd_pcm_dmaengine spd5118 iTCO_vendor_support btintel spi_nor rapl snd_pcm iwlwifi intel_cstate igc snd_timer bluetooth intel_uncore mtd snd ptp i2c_i801 pcspkr soundcore wmi_bmof cfg80211 8250_dw pps_core i2c_smbus mei_me i2c_mux rfkill mei intel_oc_wdt igen6_edac intel_pmc_core pmt_telemetry pmt_discovery acpi_tad pmt_class intel_pmc_ssram_telemetry pps_gen_tio pinctrl_alderlake pps_gen_core acpi_pad mac_hid tcp_bbr crypto_user nfnetlink zram 842_decompress 842_compress lz4hc_compress lz4_compress xfs xe intel_vsec drm_ttm_helper drm_suballoc_helper gpu_sched drm_gpuvm drm_exec drm_gpusvm_helper i915 spi_pxa2xx_platform dw_dmac spi_pxa2xx_core nvme i2c_algo_bit drm_buddy
[  184.379747]  nvme_core ttm nvme_keyring spi_intel_pci intel_gtt nvme_auth intel_lpss_pci spi_intel drm_display_helper hkdf intel_lpss cec idma64 video wmi
[  184.379766] CPU: 0 UID: 1004 PID: 7600 Comm: crun Tainted: G           OE       6.19.6-1-cachyos-server #1 NONE  3b8ffe0a07f0df715aa7edafba617e91197795c8
[  184.379774] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[  184.379775] Hardware name: AZW EQ13/EQ13, BIOS EQ13V108 04/15/2024
[  184.379779] Sched_ext: lavd_1.1.0_gc505008f_x86_64_unknown_linux_gnu (enabled+all), task: runnable_at=+0ms
[  184.379781] RIP: 0010:scx_cgroup_move_task+0xa8/0xd0
[  184.379787] Code: 48 8b 05 e3 48 3b 03 48 8b 3c 24 48 c7 80 00 03 00 00 00 00 00 00 48 c7 87 60 03 00 00 00 00 00 00 48 83 c4 08 c3 cc cc cc cc <0f> 0b 48 c7 87 60 03 00 00 00 00 00 00 c3 cc cc cc cc c3 cc cc cc
[  184.379790] RSP: 0018:ffffcc6bc8907a08 EFLAGS: 00010046
[  184.379794] RAX: ffff8b86e95cc000 RBX: ffff8b84c4163600 RCX: ffff8b85f4383600
[  184.379797] RDX: 00000001a5c7f1ff RSI: ffff8b863148da00 RDI: ffff8b84c4163600
[  184.379799] RBP: ffff8b882fb329c0 R08: ffff8b8794965340 R09: 0000000000000001
[  184.379801] R10: 0000002aede09632 R11: 0000000000000000 R12: ffff8b84c4163600
[  184.379804] R13: ffff8b882fa1cee0 R14: 0000000000000000 R15: 0000000000000000
[  184.379807] FS:  00007faebe732200(0000) GS:ffff8b887a449000(0000) knlGS:0000000000000000
[  184.379810] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  184.379813] CR2: 00007faebeb3dd80 CR3: 0000000105055003 CR4: 0000000000f72ef0
[  184.379815] PKRU: 55555554
[  184.379818] Call Trace:
[  184.379821]  <TASK>
[  184.379823]  sched_move_task+0x20f/0x2f0
[  184.379838]  cpu_cgroup_attach+0x3d/0x70
[  184.379843]  cgroup_migrate_execute+0x5d1/0x710
[  184.379850]  cgroup_update_dfl_csses+0x481/0x710
[  184.379856]  cgroup_subtree_control_write+0x442/0x560
[  184.379861]  kernfs_fop_write_iter+0x149/0x200
[  184.379867]  vfs_write+0x2ff/0x4c0
[  184.379874]  __x64_sys_write+0x70/0xe0
[  184.379879]  do_syscall_64+0x81/0x610
[  184.379887]  ? do_syscall_64+0x81/0x610
[  184.379892]  ? kmem_cache_free+0x4a7/0x4e0
[  184.379899]  ? __x64_sys_close+0x89/0xf0
[  184.379904]  ? __x64_sys_close+0x89/0xf0
[  184.379908]  ? do_syscall_64+0x81/0x610
[  184.379912]  ? __x64_sys_openat+0xfa/0x240
[  184.379917]  ? do_syscall_64+0x81/0x610
[  184.379922]  ? do_syscall_64+0x81/0x610
[  184.379926]  ? do_syscall_64+0x81/0x610
[  184.379930]  ? do_syscall_64+0x81/0x610
[  184.379934]  ? exc_page_fault+0x7e/0x1b0
[  184.379940]  entry_SYSCALL_64_after_hwframe+0x79/0x81
[  184.379945] RIP: 0033:0x7faebea453be
[  184.379987] Code: 4d 89 d8 e8 64 bb 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 <c9> c3 83 e2 39 83 fa 08 75 e7 e8 03 ff ff ff 0f 1f 00 f3 0f 1e fa
[  184.379990] RSP: 002b:00007ffe569dcee0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[  184.379994] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007faebea453be
[  184.379996] RDX: 0000000000000007 RSI: 000055b6e5a73849 RDI: 0000000000000007
[  184.379998] RBP: 00007ffe569dcef0 R08: 0000000000000000 R09: 0000000000000000
[  184.380000] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000007
[  184.380002] R13: 000055b6e5a73849 R14: 0000000000000000 R15: 000055b6e5a83360
[  184.380007]  </TASK>
[  184.380009] ---[ end trace 0000000000000000 ]---

[kakra]

This may have been fixed with kernel 6.18.22 for me, probably through this commit: torvalds/linux@0c4a59d

I do no longer see the incidents involving the browser but this still needs more testing and different workloads to be sure.

[kakra]

Looks good after some testing, with some new messages which are completely new to me and I've never seen before:

[76130.240866] Switching RCU Tasks Trace to per-CPU callback queuing.
[76130.554112] Starting switch RCU Tasks Trace to CPU-0 callback queuing.
[76130.624099] Completing switch RCU Tasks Trace to CPU-0 callback queuing.

But this doesn't seem to indicate any instability or performance problem (well, it probably indicates a performance problem because it seems to mean that RCU is rebalanced to different CPU cores, but the system felt fine).

IOW, performance feels fine and stress testing with games or browser previously triggered the problem, and it no longer does. So from my perspective, this issue could be closed as solved.