insmod: ERROR: could not insert module phoenixfs.ko: Operation not permitted

[tuninger]

Compilation was completed successfully according to the README.md documentation, but insmod fails with an error：
[100%] Inserting kernel module
sudo insmod phoenixfs.ko
insmod: ERROR: could not insert module phoenixfs.ko: Operation not permitted
make: *** [Makefile:17: insmod] Error 1
make[3]: *** [CMakeFiles/insmod.dir/build.make:71: CMakeFiles/insmod] Error 2
make[2]: *** [CMakeFiles/Makefile2:247: CMakeFiles/insmod.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:254: CMakeFiles/insmod.dir/rule] Error 2
make: *** [Makefile:195: insmod] Error 2

Logs from dmesg indicate a memory type mismatch, failure in P2P memory allocation, and an NPU initialization error，
[ 119.390808] nvfs listing GPU paths:
[ 119.390985] nvfs_pci pci device entry[4294967295] 0000:95:00:0 path:
[ 119.391121] nvfs_pci pci device entry[4294967295] 0000:2b:00:0 path:
[ 119.391190] devdrv_get_devnum num:1
[ 119.391417] npu0: bus is 95, size is 68719476736, paddr is 2cd000000000
[ 119.391450] npu_devm_memremap 1
[ 119.391453] npu->pgmap->res.start is 2cd000000000, end is 2cdfffffffff
[ 119.391577] x86/PAT: insmod:3226 conflicting memory types 2cd000000000-2ce000000000 write-combining<->uncached-minus
[ 119.391589] x86/PAT: memtype_reserve failed [mem 0x2cd000000000-0x2cdfffffffff], track uncached-minus, req uncached-minus
[ 119.443167] npu numa is 1
[ 119.443181] pci_alloc_p2pmem fail!
[ 119.443186] npu_ctrl_init error:-22

lsmod | grep nvidia
nvidia_uvm 2158592 0
nvidia_fs 274432 0
nvidia_drm 139264 0
nvidia_modeset 1732608 1 nvidia_drm
nvidia 11558912 15 nvidia_uvm,nvidia_fs,nvidia_modeset
drm_kms_helper 274432 5 ast,nvidia_drm
video 73728 1 nvidia_modeset
ecc 45056 1 nvidia
drm 765952 6 drm_kms_helper,ast,drm_shmem_helper,nvidia,nvidia_drm

lsof /dev/nvidia* -- no output

Are there any specific hardware prerequisites for this project?

[st0nie]

Can u try to add nopat into boot parameters?

[tuninger]

@st0nie ，Thank you for your reply, but the issue still persists.

[Herlly]

You can try to add drm_kms_helper to blacklist or rmmod nvidia_drm and drm_kms_helper.

[tuninger]

@Herlly Thank you for your reply,I have successfully inserted the phoenixfs module,

lsmod | grep pho
phoenixfs 188416 0

but there is an issue during testing. The specific error message is as follows:

[INFO] 2026-01-14 05:37:01,119 [run_all_benchmarks.py:29] : sudo /root/0114/phoenix/sc25/../build/bin/microbenchmark -f /mnt/gds_nvme3/home/sc25/p5800/dataset/kvcache_tensor.bin -l 10G -s 4096 -t 1 -i 1 -m read -a 0 -x 1
Assertion failure, file index :0 line :1394
Aborted
Traceback (most recent call last):
File "/root/0114/phoenix/sc25/run_all_benchmarks.py", line 440, in
eval("run_fig{}()".format(i))
File "", line 1, in
File "/root/0114/phoenix/sc25/run_all_benchmarks.py", line 127, in run_fig4
data = run_io(func_args(t, bs * 1024))
File "/root/0114/phoenix/sc25/run_all_benchmarks.py", line 111, in run_io
return run_cmd(bin_path, args)
File "/root/0114/phoenix/sc25/run_all_benchmarks.py", line 32, in run_cmd
popen = subprocess.check_output(cmdline, shell=True)
File "/root/.local/share/uv/python/cpython-3.10.12-linux-x86_64-gnu/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/root/.local/share/uv/python/cpython-3.10.12-linux-x86_64-gnu/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'sudo /root/0114/phoenix/sc25/../build/bin/microbenchmark -f /mnt/gds_nvme3/home/sc25/p5800/dataset/kvcache_tensor.bin -l 10G -s 4096 -t 1 -i 1 -m read -a 0 -x 1' returned non-zero exit status 134.

sudo /root/0114/phoenix/sc25/../build/bin/microbenchmark -f /mnt/gds_nvme3/home/sc25/p5800/dataset/kvcache_tensor.bin -l 1G -s 4096 -t 1 -i 1 -m read -a 0 -x 1
GDSOpts:
mode: 0
async: 0
file_path: /mnt/gds_nvme3/home/sc25/p5800/dataset/kvcache_tensor.bin
length: 1073741824
io_size: 4096
gpu_id: 0
num_threads: 1
xfer_mode: 1
io_depth: 1
Assertion failure, file index :0 line :1394
Aborted

sudo /root/0114/phoenix/build/bin/microbenchmark
-f /mnt/gds_nvme3/home/sc25/p5800/dataset/kvcache_tensor.bin
-l 10G -s 4096 -t 1 -i 1 -m read -a 1 -x 1
GDSOpts:
mode: 0
async: 1
file_path: /mnt/gds_nvme3/home/sc25/p5800/dataset/kvcache_tensor.bin
length: 10737418240
io_size: 4096
gpu_id: 0
num_threads: 1
xfer_mode: 1
io_depth: 1

batch_chunk_size: 10737418240
data 1async_thread

Total IO operations: 2621440
Average IO bandwidth: 44.8064 MB/s
Average IO latency: 87.1147 ns
95th percentile latency: 114.103 us
99th percentile latency: 141.795 us
99.9th percentile latency: 159.334 us

The fact that -a 1 works indicates that the GDS path itself is fine, and the crash occurs in the sync cuFileRead path of libcufile.

How can I resolve this issue? Thank you.

[Herlly]

Could you provide detailed information about the cufile.log file? This looks like the file doesn't meet GDS requirements.

[tuninger]

@Herlly ，No entries were recorded in the cufile.log.

/usr/local/cuda-12.4/gds/tools/gdsio -f /mnt/data/test_direct -d 0 -n 0 -w 16 -s 4G -i 1M -x 0
IoType: READ XferType: GPUD Threads: 16 DataSetSize: 3956736/4194304(KiB) IOSize: 1024(KiB) Throughput: 10.025792 GiB/sec, Avg_Latency: 1552.008433 usecs ops: 3864 total_time 0.376373 secs

[tuninger]

@Herlly Please find my environment details and the error message below.

(.venv) root@host:~/0114/phoenix/sc25# /usr/local/cuda-12.4/gds/tools/gdscheck -p

=====================
GDS release version: 1.9.0.20
nvidia_fs version: 2.19 libcufile version: 2.12
Platform: x86_64
ENVIRONMENT:
DRIVER CONFIGURATION:
NVMe : Supported
NVMeOF : Unsupported
SCSI : Unsupported
ScaleFlux CSD : Unsupported
NVMesh : Unsupported
DDN EXAScaler : Unsupported
IBM Spectrum Scale : Unsupported
NFS : Unsupported
BeeGFS : Unsupported
WekaFS : Unsupported
Userspace RDMA : Unsupported
--Mellanox PeerDirect : Enabled
--rdma library : Not Loaded (libcufile_rdma.so)
--rdma devices : Not configured
--rdma_device_status : Up: 0 Down: 0
CUFILE CONFIGURATION:
properties.use_compat_mode : false
properties.force_compat_mode : false
properties.gds_rdma_write_support : true
properties.use_poll_mode : false
properties.poll_mode_max_size_kb : 4
properties.max_batch_io_size : 128
properties.max_batch_io_timeout_msecs : 5
properties.max_direct_io_size_kb : 4096
properties.max_device_cache_size_kb : 131072
properties.max_device_pinned_mem_size_kb : 18014398509481980
properties.posix_pool_slab_size_kb : 4 1024 16384
properties.posix_pool_slab_count : 128 64 32
properties.rdma_peer_affinity_policy : RoundRobin
properties.rdma_dynamic_routing : 0
fs.generic.posix_unaligned_writes : false
fs.lustre.posix_gds_min_kb: 0
fs.beegfs.posix_gds_min_kb: 0
fs.weka.rdma_write_support: false
fs.gpfs.gds_write_support: false
profile.nvtx : false
profile.cufile_stats : 0
miscellaneous.api_check_aggressive : false
execution.max_io_threads : 0
execution.max_io_queue_depth : 128
execution.parallel_io : false
execution.min_io_threshold_size_kb : 4
execution.max_request_parallelism : 0
properties.force_odirect_mode : false
properties.prefer_iouring : true
GPU INFO:
GPU index 0 NVIDIA A30 bar:1 bar size (MiB):65536 supports GDS, IOMMU State: Disabled
PLATFORM INFO:
IOMMU: disabled
Nvidia Driver Info Status: Supported(Nvidia Open Driver Installed)
Cuda Driver Version Installed: 12040
Platform: Arch: x86_64(Linux 6.1.0-rc8)
Platform verification succeeded

(.venv) root@host:~/0114/phoenix/sc25# cat /etc/issue
Ubuntu 22.04.5 LTS \n \l

(.venv) root@host:~/0114/phoenix/sc25# uname -r
6.1.0-rc8

(.venv) root@host:~/0114/phoenix/sc25# nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Feb_27_16:19:38_PST_2024
Cuda compilation tools, release 12.4, V12.4.99
Build cuda_12.4.r12.4/compiler.33961263_0

(.venv) root@host:~/0114/phoenix/sc25# cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nopat iommu=off pci=realloc p2pdma=on iomem=relaxed"

(.venv) root@host:~/0114/phoenix/sc25# python run_all_benchmarks.py -a fig3
[INFO] 2026-01-27 01:23:19,053 [run_all_benchmarks.py:29] : sudo /root/0114/phoenix/sc25/../build/bin/breakdown /mnt/gds_nvme3/home/sc25/p5800/dataset/kvcache_tensor.bin 0 4 10
cuda runtime api call failed cudaSetDevice(device_id)182:cudaErrorNotSupported
EXITING program!!!
Traceback (most recent call last):
File "/root/0114/phoenix/sc25/run_all_benchmarks.py", line 442, in
eval(f"run_{args.artifact}()")
File "", line 1, in
File "/root/0114/phoenix/sc25/run_all_benchmarks.py", line 42, in run_fig3
result[t].append(run_cmd(bin_path, [file_path, t, bs, 10]))
File "/root/0114/phoenix/sc25/run_all_benchmarks.py", line 32, in run_cmd
popen = subprocess.check_output(cmdline, shell=True)
File "/root/.local/share/uv/python/cpython-3.10.12-linux-x86_64-gnu/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/root/.local/share/uv/python/cpython-3.10.12-linux-x86_64-gnu/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'sudo /root/0114/phoenix/sc25/../build/bin/breakdown /mnt/gds_nvme3/home/sc25/p5800/dataset/kvcache_tensor.bin 0 4 10' returned non-zero exit status 1.

(.venv) root@host:/0114/phoenix/sc25# sudo /root/0114/phoenix/sc25/../build/bin/breakdown /mnt/gds_nvme3/home/sc25/p5800/dataset/kvcache_tensor.bin 0 4 10
/mnt/gds_nvme3/home/sc25/p5800/dataset/kvcache_tensor.bin
io size4096Start to get breakdown
cuda runtime api call failed cudaSetDevice(device_id)182:cudaErrorNotSupported
EXITING program!!!
(.venv) root@host:/0114/phoenix/sc25# cat cufile.log -- no infos

I would appreciate it if you could take a look and help me resolve this. Thank you!

[Herlly]

error info: cuda runtime api call failed cudaSetDevice(device_id)182:cudaErrorNotSupported. I think you can first check whether the device ID in cudaSetDevice is correct. And you should remove the nopat and pci=realloc p2pdma=on iomem=relaxed parameters from the kernel grub cmdline.

[tuninger]

@Herlly ，Thank you for the reply. I’ve implemented the changes as suggested. After restarting, some scenarios are working successfully, but others are still running into errors.

cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=off"

cat /etc/cufile.json
{
"logging": { "level": "ERROR" },
"properties": {
"allow_compat_mode": false,
"force_compat_mode": false,
"allow_unsupported_fs_type": true,
"allow_generic_nvme": true,
"prefer_iouring": true,
"force_odirect_mode": true,
"allow_cross_node_access": true,
"max_direct_io_size_kb": 16384
},
"fs": {
"generic": {
"posix_unaligned_writes": false
}
},
"execution": {
"min_io_threshold_size_kb": 4
}
}

df -Th
Filesystem Type Size Used Avail Use% Mounted on
tmpfs tmpfs 101G 3.4M 101G 1% /run
/dev/sda2 ext4 878G 99G 735G 12% /
tmpfs tmpfs 504G 0 504G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda1 vfat 1.1G 6.1M 1.1G 1% /boot/efi
tmpfs tmpfs 101G 4.0K 101G 1% /run/user/1000
/dev/nvme0n1p1 xfs 12T 116G 12T 1% /mnt/gds
/dev/nvme1n1p1 xfs 12T 116G 12T 1% /mnt/gds1

/usr/local/cuda-12.4/gds/tools/gdscheck -f /mnt/gds1/home/sc25/p5800/dataset/kvcache_tensor.bin
GDS register success
generating 4k read latency matrix :
GPU 3b:00:00 : 8.34(us) read_verification: pass

root@host:/home/host# /usr/local/cuda-12.4/gds/tools/gdsio -f /mnt/gds/test_direct -I 1 -d 0 -n 0 -w 16 -s 4G -i 1M -x 0
IoType: WRITE XferType: GPUD Threads: 16 DataSetSize: 4174848/4194304(KiB) IOSize: 1024(KiB) Throughput: 5.270324 GiB/sec, Avg_Latency: 2960.721710 usecs ops: 4077 total_time 0.755446 secs
root@host:/home/host# /usr/local/cuda-12.4/gds/tools/gdsio -f /mnt/gds/test_direct -I 1 -d 0 -n 0 -w 16 -s 4G -i 1M -x 1
IoType: WRITE XferType: CPUONLY Threads: 16 DataSetSize: 4194304/4194304(KiB) IOSize: 1024(KiB) Throughput: 4.203213 GiB/sec, Avg_Latency: 2928.027100 usecs ops: 4096 total_time 0.951653 secs
root@host:/home/host#
root@host:/home/host# /usr/local/cuda-12.4/gds/tools/gdsio -f /mnt/gds/test_direct -I 0 -d 0 -n 0 -w 16 -s 4G -i 1M -x 0
IoType: READ XferType: GPUD Threads: 16 DataSetSize: 4088832/4194304(KiB) IOSize: 1024(KiB) Throughput: 11.302584 GiB/sec, Avg_Latency: 1374.902288 usecs ops: 3993 total_time 0.345002 secs
root@host:/home/host# /usr/local/cuda-12.4/gds/tools/gdsio -f /mnt/gds/test_direct -I 0 -d 0 -n 0 -w 16 -s 4G -i 1M -x 1
IoType: READ XferType: CPUONLY Threads: 16 DataSetSize: 4194304/4194304(KiB) IOSize: 1024(KiB) Throughput: 7.366537 GiB/sec, Avg_Latency: 1357.989014 usecs ops: 4096 total_time 0.542996 secs

sudo /root/0127/phoenix/sc25/../build/bin/microbenchmark -f /mnt/gds/home/sc25/p5800/dataset/kvcache_tensor.bin -l 1G -s 4096 -t 1 -i 1 -m read -a 0 -x 0 【It executes successfully, but the IO bandwidth is significantly lower than expected.】
GDSOpts:
mode: 0
async: 0
file_path: /mnt/gds/home/sc25/p5800/dataset/kvcache_tensor.bin
length: 1073741824
io_size: 4096
gpu_id: 0
num_threads: 1
xfer_mode: 0
io_depth: 1
Total IO operations: 262144
Average IO bandwidth: 1.78401 MB/s
Average IO latency: 2189.54 us
95th percentile latency: 2199.55 us
99th percentile latency: 2205.01 us
99.9th percentile latency: 2231.85 us

sudo /root/0114/phoenix/sc25/../build/bin/microbenchmark -f /mnt/gds/home/sc25/p5800/dataset/kvcache_tensor.bin -l 1G -s 4096 -t 1 -i 1 -m read -a 0 -x 1 【It executes failed】

GDSOpts:
mode: 0
async: 0
file_path: /mnt/gds/home/sc25/p5800/dataset/kvcache_tensor.bin
length: 1073741824
io_size: 4096
gpu_id: 0
num_threads: 1
xfer_mode: 1
io_depth: 1
Assertion failure, file index :0 line :1394
Aborted

cufile.log is not generating any log output

[Herlly]

Could you please provide the result about command dmesg and mount. It seems that there is something wrong with the file you tested.

[tuninger]

dmesg | grep mount
[ 23.615989] EXT4-fs (sda2): mounted filesystem with ordered data mode. Quota mode: none.
[ 24.531412] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[ 24.609063] systemd[1]: Starting Remount Root and Kernel File Systems...
[ 24.632255] EXT4-fs (sda2): re-mounted. Quota mode: none.
[ 24.642384] systemd[1]: Finished Remount Root and Kernel File Systems.
[ 27.778107] audit: type=1400 audit(1769489927.988:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=2074 comm="apparmor_parser"
[ 102.625512] XFS (nvme0n1p1): Ending clean mount
[ 102.627173] xfs filesystem being mounted at /mnt/gds supports timestamps until 2038 (0x7fffffff)
[ 119.541937] XFS (nvme1n1p1): Ending clean mount
[ 119.544038] xfs filesystem being mounted at /mnt/gds1 supports timestamps until 2038 (0x7fffffff)
[ 345.366839] XFS (nvme1n1p1): Unmounting Filesystem
[ 706.025615] XFS (nvme1n1p1): Ending clean mount
[ 706.027581] xfs filesystem being mounted at /mnt/gds1 supports timestamps until 2038 (0x7fffffff)

[Herlly]

The result of the individual "mount" command and the last 20 lines of "dmesg" output

[tuninger]

Furthermore, I've identified another problem.

root@host:~/0127/phoenix/build# sudo LD_LIBRARY_PATH=. ./bin/microbenchmark -f /mnt/gds1/home/sc25/p5800/dataset/kvcache_tensor.bin -l 1M -s 1024 -t 1 -i 1 -m read -a 0 -x 1
GDSOpts:
mode: 0
async: 0
file_path: /mnt/gds1/home/sc25/p5800/dataset/kvcache_tensor.bin
length: 1048576
io_size: 1024
gpu_id: 0
num_threads: 1
xfer_mode: 1
io_depth: 1
Total IO operations: 1024
Average IO bandwidth: 3.52691 MB/s
Average IO latency: 276.28 ns
95th percentile latency: 763.811 us
99th percentile latency: 778.575 us
99.9th percentile latency: 1198.7 us

root@host:~/0127/phoenix/build# sudo LD_LIBRARY_PATH=. ./bin/microbenchmark -f /mnt/gds1/home/sc25/p5800/dataset/kvcache_tensor.bin -l 1M -s 2048 -t 1 -i 1 -m read -a 0 -x 1
GDSOpts:
mode: 0
async: 0
file_path: /mnt/gds1/home/sc25/p5800/dataset/kvcache_tensor.bin
length: 1048576
io_size: 2048
gpu_id: 0
num_threads: 1
xfer_mode: 1
io_depth: 1
Total IO operations: 512
Average IO bandwidth: 4.75803 MB/s
Average IO latency: 409.342 ns
95th percentile latency: 767.426 us
99th percentile latency: 785.352 us
99.9th percentile latency: 3065.87 us

root@host:~/0127/phoenix/build# sudo LD_LIBRARY_PATH=. ./bin/microbenchmark -f /mnt/gds1/home/sc25/p5800/dataset/kvcache_tensor.bin -l 1M -s 4096 -t 1 -i 1 -m read -a 0 -x 1
GDSOpts:
mode: 0
async: 0
file_path: /mnt/gds1/home/sc25/p5800/dataset/kvcache_tensor.bin
length: 1048576
io_size: 4096
gpu_id: 0
num_threads: 1
xfer_mode: 1
io_depth: 1
Assertion failure, file index :0 line :1394
Aborted

[tuninger]

The result of the individual "mount" command and the last 20 lines of "dmesg" output

root@host:~/0127/phoenix/build# mount | grep /mnt/gds1
/dev/nvme1n1p1 on /mnt/gds1 type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)

root@host:~/0127/phoenix/build# dmesg | tail -n 20
[ 5573.671883] nvfs listing GPU paths:
[ 5573.671928] nvfs_pci pci device entry[4294967295] 0000:3b:00:0 path:
[ 5573.672096] nvfs_pci pci device entry[4294967295] 0000:2b:00:0 path:
[ 5573.672163] devdrv_get_devnum num:1
[ 5573.672219] npu0: bus is 3b, size is 68719476736, paddr is 24d000000000
[ 5573.672239] npu_devm_memremap 1
[ 5573.672241] npu->pgmap->res.start is 24d000000000, end is 24dfffffffff
[ 5574.189572] memmap_init_zone_device initialised 16777216 pages in 132ms
[ 5574.189578] npu numa is 0
[ 5574.189579] npu devm_memremap_pages success, addr is ff46bd8300000000
[ 5574.189934] phxfs_cdev_init success!
[ 5578.731755] memmap_init_zone_device initialised 11665408 pages in 52ms
[ 5616.963213] memmap_init_zone_device initialised 11665408 pages in 52ms
[ 5629.003338] memmap_init_zone_device initialised 11665408 pages in 56ms
[ 5639.156293] memmap_init_zone_device initialised 11665408 pages in 48ms
[ 5650.709272] memmap_init_zone_device initialised 11665408 pages in 48ms
[ 5792.585681] memmap_init_zone_device initialised 11665408 pages in 48ms
[ 5807.058696] memmap_init_zone_device initialised 11665408 pages in 52ms
[ 5811.208810] memmap_init_zone_device initialised 11665408 pages in 48ms
[ 5824.828335] memmap_init_zone_device initialised 11665408 pages in 48ms

[Herlly]

We haven't conducted any tests on the xfs file system. You could try replacing it with ext4 instead? Additionally, Direct IO has a minimum requirement for the size of I/O operations. You should avoid attempting I/O sizes below 4K.