prov/efa: Minor changes necessary for the protocol path refactor by sunkuamzn · Pull Request #12010 · ofiwg/libfabric

sunkuamzn · 2026-03-17T00:20:04Z

Prerequisite for #12002

talavr-amazon · 2026-03-18T20:53:41Z

 	/* Work arrays for efa_rdm_ope_post_send to avoid stack allocation */
 	struct efa_rdm_pke **send_pkt_entry_vec;
-	int *send_pkt_entry_size_vec;
+	int *send_pkt_entry_data_sizes;


should it not be rename send_pkt_entry_vec?

talavr-amazon · 2026-03-18T20:54:24Z

 {
 	ssize_t err;
 	int rtm_type, use_p2p;



please use zero copy in the commit message. also the commit message should explain the dependency

The pkt entries are stored in send_pkt_entry_vec, so the size should also be stored in the endpoint Also rename send_pkt_entry_size_vec to send_pkt_entry_data_sizes to avoid confusion between the two members Signed-off-by: Sai Sunku <sunkusa@amazon.com>

If the send queue is full, no point creating txe and pkes Signed-off-by: Sai Sunku <sunkusa@amazon.com>

Remove the efa_rdm_ep_alloc_txe helper and inline its logic at each call site. The function combined pool allocation, TXE construction, tagged field setup, and list insertion in a way that made it difficult to separate allocation from initialization for the new protocol interface. Move the dlist_insert_tail for ep->txe_list into efa_rdm_txe_construct so that all callers get consistent list management without duplicating the insertion call. Signed-off-by: Sai Sunku <sunkusa@amazon.com>

Move efa_rdm_ep_get_memory_alignment from efa_rdm_ep_utils.c to efa_rdm_ep.h as a static inline function. The function is small and called in the packet construction hot path, so inlining avoids function call overhead. Also move the EFA_RDM_*_ALIGNMENT constant definitions from efa.h to efa_rdm_ep.h to keep them co-located with the function that uses them. Signed-off-by: Sai Sunku <sunkusa@amazon.com>

talavr-amazon · 2026-04-15T17:38:31Z

@@ -195,7 +195,8 @@ struct efa_rdm_ep {
 	struct efa_rdm_pke **pke_vec;
 	/* Work arrays for efa_rdm_ope_post_send to avoid stack allocation */
 	struct efa_rdm_pke **send_pkt_entry_vec;


Will it be cleaner if we create a send_pkt_entry struct to wrap these 3 together?

talavr-amazon · 2026-04-15T17:39:49Z

 	/* Work arrays for efa_rdm_ope_post_send to avoid stack allocation */
 	struct efa_rdm_pke **send_pkt_entry_vec;
-	int *send_pkt_entry_size_vec;
+	size_t *send_pkt_entry_data_sizes;


nit: if they're not bundled together, maybe name it send_pkt_entry_vec_data/payload_sizes

talavr-amazon · 2026-04-15T17:40:39Z

 	ssize_t err;
 	struct efa_rdm_ope *txe;
 	struct efa_rdm_peer *peer;
+	int available_tx_pkts;


unsigned or add an assert that it's >0?

talavr-amazon · 2026-04-15T17:47:24Z

+	if (!txe)
+		return NULL;
+
+	efa_rdm_txe_construct(txe, efa_rdm_ep, peer, &msg, op, 0);


should we not check for ofi_op_tagged here?

talavr-amazon · 2026-04-15T17:50:55Z

+/*
+ * The default memory alignment
+ */
+#define EFA_RDM_DEFAULT_MEMORY_ALIGNMENT (8)


if these are place in this header and are endpoint specific values, they should be prefixed with EFA_RDM_EP_

shijin-aws · 2026-04-15T18:04:12Z

 	struct efa_rdm_pke **send_pkt_entry_vec;
-	int *send_pkt_entry_size_vec;
+	size_t *send_pkt_entry_data_sizes;
+	size_t send_pkt_entry_vec_size;


I thought we need a new member in ep only when

The member needs a persistent allocation

The member can be used in different function calls through the ep lifecycle

Is the send_pkt_entry_vec_size fitting any of them? Per this commit it seems it is only used in the efa_rdm_ope_post_send? Is it a preparation for future refactor?

If it is a prerequisite for protocol refactor (callback), I think this field should belong to a struct for protocol instead of ep

shijin-aws · 2026-04-15T18:05:41Z

 	}

+	// Handle case when there are no TX packets available
+	available_tx_pkts = ep->efa_max_outstanding_tx_ops -


Maybe better to have it as a static inline util function as it may be used by multiple places

charlesstoll reviewed Mar 17, 2026

View reviewed changes

Comment thread prov/efa/src/rdm/efa_rdm_ep_utils.c

charlesstoll reviewed Mar 17, 2026

View reviewed changes

Comment thread prov/efa/src/rdm/efa_rdm_msg.c Outdated

charlesstoll reviewed Mar 17, 2026

View reviewed changes

Comment thread prov/efa/src/rdm/efa_rdm_msg.c Outdated

sunkuamzn force-pushed the proto-callback-prereq branch from a486551 to 76b9f74 Compare March 17, 2026 21:10

sunkuamzn requested a review from a team March 18, 2026 01:20

talavr-amazon reviewed Mar 18, 2026

View reviewed changes

Comment thread prov/efa/src/rdm/efa_rdm_ep.h Outdated

talavr-amazon reviewed Mar 18, 2026

View reviewed changes

Comment thread prov/efa/src/rdm/efa_rdm_msg.c Outdated

talavr-amazon reviewed Mar 18, 2026

View reviewed changes

Comment thread prov/efa/src/rdm/efa_rdm_msg.c Outdated

talavr-amazon reviewed Mar 18, 2026

View reviewed changes

Comment thread prov/efa/src/rdm/efa_rdm_atomic.c

talavr-amazon requested changes Mar 18, 2026

View reviewed changes

sunkuamzn force-pushed the proto-callback-prereq branch 5 times, most recently from ebd47a2 to afaf415 Compare April 10, 2026 23:38

sunkuamzn added 2 commits April 11, 2026 00:36

prov/efa: Check if the send queue is full earlier

9775660

If the send queue is full, no point creating txe and pkes Signed-off-by: Sai Sunku <sunkusa@amazon.com>

sunkuamzn force-pushed the proto-callback-prereq branch from afaf415 to 88ed8af Compare April 11, 2026 01:26

sunkuamzn added 2 commits April 11, 2026 01:57

sunkuamzn force-pushed the proto-callback-prereq branch from 88ed8af to 7d9e360 Compare April 11, 2026 01:58

sunkuamzn mentioned this pull request Apr 11, 2026

prov/efa: Introduce new protocol interface and implement it for the TX path #12002

Open

talavr-amazon reviewed Apr 15, 2026

View reviewed changes

talavr-amazon approved these changes Apr 15, 2026

View reviewed changes

talavr-amazon reviewed Apr 15, 2026

View reviewed changes

talavr-amazon approved these changes Apr 15, 2026

View reviewed changes

talavr-amazon reviewed Apr 15, 2026

View reviewed changes

talavr-amazon approved these changes Apr 15, 2026

View reviewed changes

shijin-aws reviewed Apr 15, 2026

View reviewed changes

Conversation

sunkuamzn commented Mar 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

talavr-amazon Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

talavr-amazon Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

talavr-amazon Mar 18, 2026 •

edited

Loading

talavr-amazon Apr 15, 2026 •

edited

Loading