Commit 6bc8d265 authored Jul 13, 2023 by Alexey Marchuk Committed by Tomasz Zawadzki Aug 31, 2023

nvmf/rdma: Clarify max_read_depth calculation



Max Inbound RDMA_READ Queue Depth (IRD) is sent by host
in responder_resources. Target must limit its ORD
(Outbound RDMA_READ Queue Depth) so it doesn't exceed
initiator's IRD.
Refer to chapter 3.1.4 Fabric Dependent Settings of
RDMA transport specification

When connecting, intiator must set responder_resources
to max outstanding RDMA_READ operations supported by
local HCA as a target (or max num of incoming operations) -
that is max_qp_rd_atom.
When accepting, kernel cm module sets
event.initiator_depth=req.responder_resources (swap)
and event.responder_resources=req.intiator_depth (swap)
see cm_req_handler() in drivers/infiniband/core/cm.c and
rdma_get_cm_event man page. When accepting, target must
limit initiator_depth (aka responder_resouces set by
intiator aka incoming RDMA_READ operations supported
by intiator) by local HCA max_qp_init_rd_atom. 0 value is
not acceptable according to nvmf spec.
At the same time, event.responder_resources (intiator_depth
set by intitiator or number of incoming RDMA_READ operations
requested by intiator which target must support) must be 0
since intiator is not allowed to issue RDMA operations
according to nvmf spec.

Signed-off-by: Alexey Marchuk <alexeymar@nvidia.com>
Change-Id: I66aa015f808471d1d055d9553b8adb8350671f83
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/19118


Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <jim.harris@gmail.com>

parent bacf7ec2

lib/nvmf/rdma.c

+19 −2

Original line number	Diff line number	Diff line
		@@ -1298,9 +1298,26 @@ nvmf_rdma_connect(struct spdk_nvmf_transport transport, struct rdma_cm_event e
		SPDK_DEBUGLOG(rdma,
		"Host (Initiator) NIC Max Incoming RDMA R/W operations: %d Max Outgoing RDMA R/W operations: %d\n",
		rdma_param->initiator_depth, rdma_param->responder_resources);
		if (rdma_param->initiator_depth > 0) {
		max_read_depth = spdk_min(max_read_depth, rdma_param->initiator_depth);
		/* from man3 rdma_get_cm_event
		* responder_resources - Specifies the number of responder resources that is requested by the recipient.
		* The responder_resources field must match the initiator depth specified by the remote node when running
		* the rdma_connect and rdma_accept functions. */
		if (rdma_param->responder_resources != 0) {
		SPDK_ERRLOG("Host (Initiator) is not allowed to use RDMA operations (responder_resources %u)\n",
		rdma_param->responder_resources);
		nvmf_rdma_event_reject(event->id, SPDK_NVMF_RDMA_ERROR_INVALID_ORD);
		return -1;
		}
		/* from man3 rdma_get_cm_event
		* initiator_depth - Specifies the maximum number of outstanding RDMA read operations that the recipient holds.
		* The initiator_depth field must match the responder resources specified by the remote node when running
		* the rdma_connect and rdma_accept functions. */
		if (rdma_param->initiator_depth == 0) {
		SPDK_ERRLOG("Host (Initiator) doesn't support RDMA_READ or atomic operations\n");
		nvmf_rdma_event_reject(event->id, SPDK_NVMF_RDMA_ERROR_INVALID_IRD);
		return -1;
		}
		max_read_depth = spdk_min(max_read_depth, rdma_param->initiator_depth);

		/* Finally check for the host software requested values, which are
		* optional. */