Commit ceb07eb8 authored by Ziye Yang's avatar Ziye Yang Committed by Tomasz Zawadzki
Browse files

nvme/tcp: Fix send_cb and recv pdu function contention when there is R2T. 

When using uring socket, we see following assert

nvme_tcp.c:1018: nvme_tcp_capsule_resp_hdr_handle:
Assertion `tcp_req->state == NVME_TCP_REQ_ACTIVE' failed.
Detailed info is in

https://ci.spdk.io/results/autotest-per-patch/builds/19205/archive/nvmf-tcp-vg-autotest/build.log



We face this issue, because there is also code execution ordering
between "sending callback function" and "pdu receving function".
We did not find it in physical machine testing, but finding it
in vagrant machine in CI.

Signed-off-by: default avatarZiye Yang <ziye.yang@intel.com>
Change-Id: I5eb241d564c0fc42ce0601b7c85999a2550f0de3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3046


Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Reviewed-by: default avatarBen Walker <benjamin.walker@intel.com>
Reviewed-by: default avatarShuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
parent c833729b

lib/nvme/nvme_tcp.c

+6 −1
Original line number Diff line number Diff line
@@ -505,6 +505,7 @@ static inline void 
nvme_tcp_req_put_safe(struct nvme_tcp_req *tcp_req)

{

	if (tcp_req->ordering.send_ack && tcp_req->ordering.data_recv) {

		assert(tcp_req->state == NVME_TCP_REQ_ACTIVE);

		assert(tcp_req->tqpair != NULL);

		nvme_tcp_req_put(tcp_req->tqpair, tcp_req);

	}
@@ -1015,7 +1016,6 @@ nvme_tcp_capsule_resp_hdr_handle(struct nvme_tcp_qpair *tqpair, struct nvme_tcp_ 
	}



	assert(tcp_req->req != NULL);

	assert(tcp_req->state == NVME_TCP_REQ_ACTIVE);

	nvme_tcp_req_complete(tcp_req->req, &cpl);

	(*reaped)++;
@@ -1122,12 +1122,15 @@ nvme_tcp_qpair_h2c_data_send_complete(void *cb_arg) 


	assert(tcp_req != NULL);



	tcp_req->ordering.send_ack = 1;

	if (tcp_req->r2tl_remain) {

		nvme_tcp_send_h2c_data(tcp_req);

	} else {

		assert(tcp_req->active_r2ts > 0);

		tcp_req->active_r2ts--;

		tcp_req->state = NVME_TCP_REQ_ACTIVE;

		/* Need also call this function to free the resource */

		nvme_tcp_req_put_safe(tcp_req);

	}

}
@@ -1139,6 +1142,8 @@ nvme_tcp_send_h2c_data(struct nvme_tcp_req *tcp_req) 
	struct spdk_nvme_tcp_h2c_data_hdr *h2c_data;

	uint32_t plen, pdo, alignment;



	/* Reinit the send_ack */

	tcp_req->ordering.send_ack = 0;

	rsp_pdu = tcp_req->send_pdu;

	memset(rsp_pdu, 0, sizeof(*rsp_pdu));

	h2c_data = &rsp_pdu->hdr.h2c_data;