NIC: Intel X553 10 GbE SFP+
OS: Ubuntu 24.04.1 LTS
Kernel: 6.8.0-90-generic
CPU: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz
The steps to reproduce:
1, unbind X553 interfaces from ixgbe and bind them to vfio-pci.
2, unbind them from vfio-pci and bind them back to ixgbe, then bring up the link with "ip link set up" and check the link state.
3, repeat the steps above.
after running about 21774 times, ixgbe crashes as below:
[282585.972997] ixgbe 0000:03:00.0 enp3s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[282586.011523] ixgbe 0000:03:00.1 enp3s0f1: detected SFP+: 6
[282586.525011] ixgbe 0000:03:00.1 enp3s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[282586.985198] ixgbe 0000:03:00.0: removed PHC on enp3s0f0
[282587.166597] ixgbe 0000:03:00.1: removed PHC on enp3s0f1
[282588.835998] ixgbe 0000:03:00.0 0000:03:00.0 (uninitialized): Warning: no configuration for board #43552
[282588.836023] ixgbe 0000:03:00.0 0000:03:00.0 (uninitialized): Using defaults for all values
[282588.836047] BUG: unable to handle page fault for address: ffffffffc11fe000
[282588.843162] #PF: supervisor read access in kernel mode
[282588.848530] #PF: error_code(0x0000) - not-present page
[282588.853899] PGD 3641067 P4D 3641067 PUD 3643067 PMD 110315067 PTE 0
[282588.860415] Oops: 0000 [#1] PREEMPT SMP NOPTI
[282588.865006] CPU: 3 PID: 2721133 Comm: python3 Tainted: G OE 6.8.0-90-generic #91-Ubuntu
[282588.874554] Hardware name: Silicom 80500-0214-E32/80300-0214-G02-SL00A, BIOS CORDOBA-03.00.00.01-SLt 2022-06-06
[282588.884875] RIP: 0010:ixgbe_check_options+0x128/0x1450 [ixgbe]
[282588.891114] Code: 25 b9 99 c3 00 89 45 88 0f 82 d8 07 00 00 85 c0 0f 85 ed 06 00 00 81 a3 24 02 00 00 7f ff ff ff 49 83 fd 21 0f 83 69 11 00 00 <42> 8b 04 ad 80 37 1d c1 8b 93 38 13 00 00 89 45 88 83 fa 03 0f 87
[282588.910126] RSP: 0018:ffffceafc5143798 EFLAGS: 00010246
[282588.915587] RAX: 0000000000000000 RBX: ffff8c8adbf009e0 RCX: 0000000000000000
[282588.922956] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[282588.930325] RBP: ffffceafc5143820 R08: 0000000000000000 R09: 0000000000000000
[282588.937695] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000aa20
[282588.945064] R13: 000000000000aa20 R14: ffffceafc51437a8 R15: ffff8c8adbf00000
[282588.952435] FS: 000074c64878a080(0000) GS:ffff8c8daf780000(0000) knlGS:0000000000000000
[282588.960764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[282588.966742] CR2: ffffffffc11fe000 CR3: 00000001092fa000 CR4: 00000000003506f0
[282588.974112] Call Trace:
[282588.976790]
[282588.979124] ? usleep_range_state+0x65/0xa0
[282588.983553] ixgbe_probe+0x893/0x2770 [ixgbe]
[282588.988285] local_pci_probe+0x47/0xb0
[282588.992278] pci_call_probe+0x55/0x1a0
[282588.996262] pci_device_probe+0x84/0x120
[282589.000423] really_probe+0x1c7/0x410
[282589.004325] __driver_probe_device+0x8c/0x180
[282589.008922] device_driver_attach+0x55/0xd0
[282589.013346] bind_store+0x77/0xd0
[282589.016898] drv_attr_store+0x24/0x50
[282589.020795] sysfs_kf_write+0x3e/0x60
[282589.024697] kernfs_fop_write_iter+0x14f/0x1e0
[282589.029379] vfs_write+0x2a8/0x480
[282589.033022] ksys_write+0x73/0x100
[282589.036661] __x64_sys_write+0x19/0x30
[282589.040643] x64_sys_call+0x7e/0x25a0
[282589.044555] do_syscall_64+0x7f/0x180
[282589.048442] ? ksys_lseek+0x80/0xd0
[282589.052153] ? arch_exit_to_user_mode_prepare.isra.0+0x1a/0xe0
[282589.058208] ? syscall_exit_to_user_mode+0x43/0x1e0
[282589.063313] ? do_syscall_64+0x8c/0x180
[282589.067377] ? _copy_to_user+0x25/0x50
[282589.071350] ? cp_new_stat+0x143/0x180
[282589.075323] ? generic_file_llseek+0x24/0x40
[282589.079818] ? kernfs_fop_llseek+0x70/0xb0
[282589.084137] ? ksys_lseek+0x80/0xd0
[282589.087848] ? generic_file_llseek+0x24/0x40
[282589.092339] ? kernfs_fop_llseek+0x70/0xb0
[282589.096659] ? ksys_lseek+0x80/0xd0
[282589.100369] ? arch_exit_to_user_mode_prepare.isra.0+0x1a/0xe0
[282589.106424] ? syscall_exit_to_user_mode+0x43/0x1e0
[282589.111523] ? do_syscall_64+0x8c/0x180
[282589.115581] ? irqentry_exit+0x43/0x50
[282589.119552] entry_SYSCALL_64_after_hwframe+0x78/0x80
[282589.124827] RIP: 0033:0x74c64851c5a4
[282589.128642] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d a5 ea 0e 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
[282589.147635] RSP: 002b:00007ffe8f9de548 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[282589.155427] RAX: ffffffffffffffda RBX: 000074c648789ff8 RCX: 000074c64851c5a4
[282589.162782] RDX: 000000000000000c RSI: 000000002accfbd0 RDI: 0000000000000003
[282589.170140] RBP: 00007ffe8f9de590 R08: 0000000000000000 R09: 0000000000000000
[282589.177497] R10: 0000000000000003 R11: 0000000000000202 R12: 000000000000000c
[282589.184853] R13: 0000000000000003 R14: 000000002accfbd0 R15: 0000000000ba5748
[282589.192213]
[282589.194617] Modules linked in: ixgbe(OE) tls vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd qrtr binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common pnd2_edac ath10k_pci x86_pkg_temp_thermal intel_powerclamp ath10k_core coretemp ath cdc_mbim cdc_wdm cdc_ncm option kvm_intel mac80211 silicom_platform cdc_ether usb_wwan cmdlinepart usbserial usbnet led_class_multicolor cfg80211 kvm spi_nor mii ee1004 libarc4 qat_c3xxx mtd irqbypass intel_qat spi_intel_pci intel_th_gth spi_intel intel_th_pci intel_th rapl i2c_i801 crc8 authenc tpm_infineon i2c_ismt intel_cstate lpc_ich i2c_smbus mac_hid dm_multipath msr parport_pc ppdev lp parport efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 uas usb_storage crct10dif_pclmul crc32_pclmul polyval_generic ghash_clmulni_intel xfrm_algo sha256_ssse3 sdhci_pci xhci_pci sha1_ssse3 dca cqhci igc sdhci mdio xhci_pci_renesas pinctrl_denverton
[282589.194743] aesni_intel crypto_simd cryptd [last unloaded: iqvlinux(OE)]
[282589.292041] CR2: ffffffffc11fe000
[282589.295578] ---[ end trace 0000000000000000 ]---
[282590.734333] RIP: 0010:ixgbe_check_options+0x128/0x1450 [ixgbe]
[282590.740596] Code: 25 b9 99 c3 00 89 45 88 0f 82 d8 07 00 00 85 c0 0f 85 ed 06 00 00 81 a3 24 02 00 00 7f ff ff ff 49 83 fd 21 0f 83 69 11 00 00 <42> 8b 04 ad 80 37 1d c1 8b 93 38 13 00 00 89 45 88 83 fa 03 0f 87
[282590.759611] RSP: 0018:ffffceafc5143798 EFLAGS: 00010246
[282590.765075] RAX: 0000000000000000 RBX: ffff8c8adbf009e0 RCX: 0000000000000000
[282590.772442] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[282590.779812] RBP: ffffceafc5143820 R08: 0000000000000000 R09: 0000000000000000
[282590.787183] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000aa20
[282590.794553] R13: 000000000000aa20 R14: ffffceafc51437a8 R15: ffff8c8adbf00000
[282590.801922] FS: 000074c64878a080(0000) GS:ffff8c8daf780000(0000) knlGS:0000000000000000
[282590.810248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[282590.816226] CR2: ffffffffc11fe000 CR3: 00000001092fa000 CR4: 00000000003506f0
[282590.823596] note: python3[2721133] exited with irqs disabled
NIC: Intel X553 10 GbE SFP+
OS: Ubuntu 24.04.1 LTS
Kernel: 6.8.0-90-generic
CPU: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz
The steps to reproduce:
1, unbind X553 interfaces from ixgbe and bind them to vfio-pci.
2, unbind them from vfio-pci and bind them back to ixgbe, then bring up the link with "ip link set up" and check the link state.
3, repeat the steps above.
after running about 21774 times, ixgbe crashes as below:
[282585.972997] ixgbe 0000:03:00.0 enp3s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[282586.011523] ixgbe 0000:03:00.1 enp3s0f1: detected SFP+: 6
[282586.525011] ixgbe 0000:03:00.1 enp3s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[282586.985198] ixgbe 0000:03:00.0: removed PHC on enp3s0f0
[282587.166597] ixgbe 0000:03:00.1: removed PHC on enp3s0f1
[282588.835998] ixgbe 0000:03:00.0 0000:03:00.0 (uninitialized): Warning: no configuration for board #43552
[282588.836023] ixgbe 0000:03:00.0 0000:03:00.0 (uninitialized): Using defaults for all values
[282588.836047] BUG: unable to handle page fault for address: ffffffffc11fe000
[282588.843162] #PF: supervisor read access in kernel mode
[282588.848530] #PF: error_code(0x0000) - not-present page
[282588.853899] PGD 3641067 P4D 3641067 PUD 3643067 PMD 110315067 PTE 0
[282588.860415] Oops: 0000 [#1] PREEMPT SMP NOPTI
[282588.865006] CPU: 3 PID: 2721133 Comm: python3 Tainted: G OE 6.8.0-90-generic #91-Ubuntu
[282588.874554] Hardware name: Silicom 80500-0214-E32/80300-0214-G02-SL00A, BIOS CORDOBA-03.00.00.01-SLt 2022-06-06
[282588.884875] RIP: 0010:ixgbe_check_options+0x128/0x1450 [ixgbe]
[282588.891114] Code: 25 b9 99 c3 00 89 45 88 0f 82 d8 07 00 00 85 c0 0f 85 ed 06 00 00 81 a3 24 02 00 00 7f ff ff ff 49 83 fd 21 0f 83 69 11 00 00 <42> 8b 04 ad 80 37 1d c1 8b 93 38 13 00 00 89 45 88 83 fa 03 0f 87
[282588.910126] RSP: 0018:ffffceafc5143798 EFLAGS: 00010246
[282588.915587] RAX: 0000000000000000 RBX: ffff8c8adbf009e0 RCX: 0000000000000000
[282588.922956] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[282588.930325] RBP: ffffceafc5143820 R08: 0000000000000000 R09: 0000000000000000
[282588.937695] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000aa20
[282588.945064] R13: 000000000000aa20 R14: ffffceafc51437a8 R15: ffff8c8adbf00000
[282588.952435] FS: 000074c64878a080(0000) GS:ffff8c8daf780000(0000) knlGS:0000000000000000
[282588.960764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[282588.966742] CR2: ffffffffc11fe000 CR3: 00000001092fa000 CR4: 00000000003506f0
[282588.974112] Call Trace:
[282588.976790]
[282588.979124] ? usleep_range_state+0x65/0xa0
[282588.983553] ixgbe_probe+0x893/0x2770 [ixgbe]
[282588.988285] local_pci_probe+0x47/0xb0
[282588.992278] pci_call_probe+0x55/0x1a0
[282588.996262] pci_device_probe+0x84/0x120
[282589.000423] really_probe+0x1c7/0x410
[282589.004325] __driver_probe_device+0x8c/0x180
[282589.008922] device_driver_attach+0x55/0xd0
[282589.013346] bind_store+0x77/0xd0
[282589.016898] drv_attr_store+0x24/0x50
[282589.020795] sysfs_kf_write+0x3e/0x60
[282589.024697] kernfs_fop_write_iter+0x14f/0x1e0
[282589.029379] vfs_write+0x2a8/0x480
[282589.033022] ksys_write+0x73/0x100
[282589.036661] __x64_sys_write+0x19/0x30
[282589.040643] x64_sys_call+0x7e/0x25a0
[282589.044555] do_syscall_64+0x7f/0x180
[282589.048442] ? ksys_lseek+0x80/0xd0
[282589.052153] ? arch_exit_to_user_mode_prepare.isra.0+0x1a/0xe0
[282589.058208] ? syscall_exit_to_user_mode+0x43/0x1e0
[282589.063313] ? do_syscall_64+0x8c/0x180
[282589.067377] ? _copy_to_user+0x25/0x50
[282589.071350] ? cp_new_stat+0x143/0x180
[282589.075323] ? generic_file_llseek+0x24/0x40
[282589.079818] ? kernfs_fop_llseek+0x70/0xb0
[282589.084137] ? ksys_lseek+0x80/0xd0
[282589.087848] ? generic_file_llseek+0x24/0x40
[282589.092339] ? kernfs_fop_llseek+0x70/0xb0
[282589.096659] ? ksys_lseek+0x80/0xd0
[282589.100369] ? arch_exit_to_user_mode_prepare.isra.0+0x1a/0xe0
[282589.106424] ? syscall_exit_to_user_mode+0x43/0x1e0
[282589.111523] ? do_syscall_64+0x8c/0x180
[282589.115581] ? irqentry_exit+0x43/0x50
[282589.119552] entry_SYSCALL_64_after_hwframe+0x78/0x80
[282589.124827] RIP: 0033:0x74c64851c5a4
[282589.128642] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d a5 ea 0e 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
[282589.147635] RSP: 002b:00007ffe8f9de548 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[282589.155427] RAX: ffffffffffffffda RBX: 000074c648789ff8 RCX: 000074c64851c5a4
[282589.162782] RDX: 000000000000000c RSI: 000000002accfbd0 RDI: 0000000000000003
[282589.170140] RBP: 00007ffe8f9de590 R08: 0000000000000000 R09: 0000000000000000
[282589.177497] R10: 0000000000000003 R11: 0000000000000202 R12: 000000000000000c
[282589.184853] R13: 0000000000000003 R14: 000000002accfbd0 R15: 0000000000ba5748
[282589.192213]
[282589.194617] Modules linked in: ixgbe(OE) tls vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd qrtr binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common pnd2_edac ath10k_pci x86_pkg_temp_thermal intel_powerclamp ath10k_core coretemp ath cdc_mbim cdc_wdm cdc_ncm option kvm_intel mac80211 silicom_platform cdc_ether usb_wwan cmdlinepart usbserial usbnet led_class_multicolor cfg80211 kvm spi_nor mii ee1004 libarc4 qat_c3xxx mtd irqbypass intel_qat spi_intel_pci intel_th_gth spi_intel intel_th_pci intel_th rapl i2c_i801 crc8 authenc tpm_infineon i2c_ismt intel_cstate lpc_ich i2c_smbus mac_hid dm_multipath msr parport_pc ppdev lp parport efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 uas usb_storage crct10dif_pclmul crc32_pclmul polyval_generic ghash_clmulni_intel xfrm_algo sha256_ssse3 sdhci_pci xhci_pci sha1_ssse3 dca cqhci igc sdhci mdio xhci_pci_renesas pinctrl_denverton
[282589.194743] aesni_intel crypto_simd cryptd [last unloaded: iqvlinux(OE)]
[282589.292041] CR2: ffffffffc11fe000
[282589.295578] ---[ end trace 0000000000000000 ]---
[282590.734333] RIP: 0010:ixgbe_check_options+0x128/0x1450 [ixgbe]
[282590.740596] Code: 25 b9 99 c3 00 89 45 88 0f 82 d8 07 00 00 85 c0 0f 85 ed 06 00 00 81 a3 24 02 00 00 7f ff ff ff 49 83 fd 21 0f 83 69 11 00 00 <42> 8b 04 ad 80 37 1d c1 8b 93 38 13 00 00 89 45 88 83 fa 03 0f 87
[282590.759611] RSP: 0018:ffffceafc5143798 EFLAGS: 00010246
[282590.765075] RAX: 0000000000000000 RBX: ffff8c8adbf009e0 RCX: 0000000000000000
[282590.772442] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[282590.779812] RBP: ffffceafc5143820 R08: 0000000000000000 R09: 0000000000000000
[282590.787183] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000aa20
[282590.794553] R13: 000000000000aa20 R14: ffffceafc51437a8 R15: ffff8c8adbf00000
[282590.801922] FS: 000074c64878a080(0000) GS:ffff8c8daf780000(0000) knlGS:0000000000000000
[282590.810248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[282590.816226] CR2: ffffffffc11fe000 CR3: 00000001092fa000 CR4: 00000000003506f0
[282590.823596] note: python3[2721133] exited with irqs disabled