BUG/MEDIUM: mux-h1: Don't set MSG_MORE on bodyless responses forwarded to client

When h1_snd_buf() inherits the CO_SFL_MSG_MORE flag from the upper layer, it
unconditionally propagates it to H1C_F_CO_MSG_MORE, which eventually sets
MSG_MORE on the sendmsg() call. For bodyless responses (HEAD, 204, 304), this
causes the kernel to cork the TCP connection for ~200ms waiting for body data
that will never be sent.

With an H1 frontend and H2 backend, this adds ~200ms of latency to many or
all bodyless responses. The 200ms corresponds to the kernel's tcp_cork_time
default. H1 backends are less affected because h1_postparse_res_hdrs() sets
HTX_FL_EOM during header parsing for bodyless responses, but H2 backends
frequently deliver the end-of-stream signal in a separate scheduling round,
leaving htx_expect_more() returning TRUE when headers are first forwarded.

The fix guards H1C_F_CO_MSG_MORE so it is only set when the connection is a
backend (H1C_F_IS_BACK) or the response is not bodyless
(!H1S_F_BODYLESS_RESP). This ensures bodyless responses on the front
connection are sent immediately without corking.

This should be backported to all stable branches.

Co-developed-by: Billy Campoli <bcampoli@meta.com>
Co-developed-by: Chandan Avdhut <cavdhut@meta.com>
Co-developed-by: Neel Raja <neelraja@meta.com

Author: codyohl

src/mux_h1.c

@@ -4885,8 +4885,14 @@ static size_t h1_snd_buf(struct stconn *sc, struct buffer *buf, size_t count, in
 
 	/* Inherit some flags from the upper layer */
 	h1c->flags &= ~(H1C_F_CO_MSG_MORE|H1C_F_CO_STREAMER);
-	if (flags & CO_SFL_MSG_MORE)
-		h1c->flags |= H1C_F_CO_MSG_MORE;
+	if (flags & CO_SFL_MSG_MORE) {
+		/* Don't set H1C_F_CO_MSG_MORE when sending a bodyless response to client.
+		 * We must do that if the response is not finished, regardless it a bodyless
+		 * response, to be sure to send it ASAP.
+		 */
+		if ((h1c->flags & H1C_F_IS_BACK) || !(h1s->flags & H1S_F_BODYLESS_RESP))
+			h1c->flags |= H1C_F_CO_MSG_MORE;
+	}
 	if (flags & CO_SFL_STREAMER)
 		h1c->flags |= H1C_F_CO_STREAMER;