Skip to content

[BUG] InfiniStore returns Unexpected opcode: 129 #197

@HonkitMaiStorage

Description

@HonkitMaiStorage

Describe the bug
After setting up the InfiniStore server, I tried to run the infinistore/example/client.py to test whether the setup is successful. Then, the error encountered here is

[2026-01-08: 10:08:18.897] [infini] [info] Connecting to <server-ip>
[2026-01-08: 10:08:18.899] [infini] [info] found device irdma0
[2026-01-08: 10:08:18.900] [infini] [warning] [../rdma.cpp:100] RoCE choose user specified gid index 1 
[2026-01-08: 10:08:18.900] [infini] [info] gid index 1, gid <irdma0-ip>
[2026-01-08: 10:08:18.157] [infini] [debug] remote rdma_conn_info: psn: 0, qpn: 92, gid: 10.102.196.102, enum mtu: 3
[2026-01-08: 10:08:18.157] [infini] [debug] local rdma_conn_info: psn: 0, qpn: 91, gid: 10.102.196.102, enum mtu: 3 rdma connection: cpu -> cuda: 0
[2026-01-08: 10:08:18.404] [infini] [info] register mr done for base_ptr : 97375080146304
[2026-01-08: 10:08:18.405] [infini] [debug] read cache/allocated/commit msg request send 97374976098784
[2026-01-08: 10:08:18.405] [infini] [error] [../libinfinistoer.cpp:164] Unexpected opcode 129

The script above is run on the same node as the InfiniStore server, meaning the server-ip is actually the localhost IP. Then, to verify the RDMA connection is successful, ib_write_bw is also run and the benchmark results has the appropriate output as well. Therefore, the RDMA connection does not have any issue.

After the client has completed the request, the server log may/may not show the following log (Invalid header)

Image

To Reproduce

Install the InfiniStore by building from source

apt install libuv1-dev
apt install libflatbuffers-dev
apt install libspdlog-dev libfmt-dev
apt install ibverbs-utils libibverbs-dev
apt install libboost-dev libboost-stacktrace-dev
pip install --no-build-isolation -e .
pip install pre-commit
pre-commit install

Then, set up the InfiniStore server using

infinistore --manage-port 8088 --service-port 12345 --dev-name irdma0 --hint-gid-index 1 --link-type Ethernet --prealloc-size 2

Afterwards, run the test script provided in this repository after pointing the IP to the correct IP Address.

python3 infinistore/example/client.py

System info

  • OS: Ubuntu 22.04
  • GPU count and types: 1 machine with 8 * 6000 Ada
  • Python version: 3.10.12
  • InfiniStore version: pulled latest at main branch

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions