Facing an error at rdmaFetchAndAddBoundary function

[juyoungbang]

Hi.
I've been trying to run the benchmark of DEFT and faced some error.

Basically, it's causing segmentation fault during the early insertion process.
So far, I figured out that it's caused from the Operation.cpp/rdmaFectchAndAddBoundary function.

Here's the status of the code when DEFT ends with segmentation fault.

My RDMA driver is MLNX_OFED_LINUX-4.9-5.1.0.0 and RDMA firmware version is 16.32.1010 which seems to have no problem.
I'm wondering whether you are familiar with such error.

[starkwj]

From the screenshot, the RDMA connection is not established correctly (the qp is null and remoteRKey is 0, they are not initialized).

You may refer to this to start (https://github.com/thustorage/deft?tab=readme-ov-file#getting-started). I haven't touched this code for a while and forgot some details, but i suggest first checking if the memcached are correct.
If the problem is not on memcached, you may check the initialization of RDMA connection (https://github.com/thustorage/deft/blob/main/src/dsm_client.cpp#L17). (E.g., did you set a large thread count for a client that exceeds the macro MAX_APP_THREAD. You can modify the macro value.)

[juyoungbang]

Thanks for your reply.
Though, what made me confusing is that the insertion is done properly up to 500000 keys then suddenly such error arises.

So after looking into it deeply, I noticed that the workload I'm using is causing the new problem.
I'm currently trying to run my own workload which is generated via YCSB, which is not quite an exceptional workload.

Moreover, I observed that the error occurs when the original YCSB load workload is executed
while the error disappears when executing the workload after sorting it in key ascending order.

Not sure if this will also help, but the error also occurs when executing the workload after sorting it in key descending order.

[starkwj]

Hi.
Does this happen when loading keys with multiple threads? That seems like a concurrency bug. It may cause this problem when loading data from an empty tree with a large number of concurrent threads, where many concurrent write and split operations happen. I have no idea what cause this bug and it may be tricky to fix this bug. You can try with a simple workround to bypass this problem, preloading some keys (e.g., 100K or more) with a single thread before performing the actual workload.

[juyoungbang]

At first I also thought of it as a concurrency issue,
but the same issue is happening on 1CN 1MN and single threaded environments;(

Anyway thank you and I'll leave a comment if the problem is somehow solved.