Commit 2470b995 authored by sijie.sun's avatar sijie.sun Committed by Tomasz Zawadzki
Browse files

nvmf/rdma: fix rdma device hotplug in nvmf 



1. During the execution of nvmf_rdma_disconnect, there is a possibility
that rdma_destroy_id is called synchronously before the acknowledgment
of the corresponding connection management (cm) event. As
rdma_destroy_id needs to wait for all event acknowledgments, this can
lead to a potential deadlock in the cm event handler thread. To avoid
this, we now ensure that the cm event is acknowledged before invoking
nvmf_rdma_disconnect.

2. In the scenario where the retry ports are removed by the user while
waiting for a new RDMA device, we should halt the process of rescanning
for new InfiniBand (IB) devices since there will be no retry ports
available. However, it's important to note that if the user attempts
to listen on the same port again, it may fail since the new IB devices
are not yet initialized. To address this, we now perform a complete
device rescan whenever a listen operation fails.

Change-Id: I62bd34a32bc3ba183fd02303fdb48c66f9e0a22d
Signed-off-by: default avatarsijie.sun <sijie.sun@smartx.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/19682


Reviewed-by: default avatarJim Harris <jim.harris@gmail.com>
Community-CI: Mellanox Build Bot
Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: default avatarAleksey Marchuk <alexeymar@nvidia.com>
parent c8a77425

lib/nvmf/rdma.c

+7 −3
Original line number Diff line number Diff line
@@ -2945,6 +2945,7 @@ nvmf_rdma_listen(struct spdk_nvmf_transport *transport, const struct spdk_nvme_t 
			    port->id->verbs);

		rdma_destroy_id(port->id);

		free(port);

		nvmf_rdma_rescan_devices(rtransport);

		return -EINVAL;

	}
@@ -3351,7 +3352,7 @@ nvmf_rdma_destroy_drained_qpair(struct spdk_nvmf_rdma_qpair *rqpair) 
}



static int

nvmf_rdma_disconnect(struct rdma_cm_event *evt)

nvmf_rdma_disconnect(struct rdma_cm_event *evt, bool *event_acked)

{

	struct spdk_nvmf_qpair		*qpair;

	struct spdk_nvmf_rdma_qpair	*rqpair;
@@ -3367,6 +3368,9 @@ nvmf_rdma_disconnect(struct rdma_cm_event *evt) 
		return -1;

	}



	rdma_ack_cm_event(evt);

	*event_acked = true;



	rqpair = SPDK_CONTAINEROF(qpair, struct spdk_nvmf_rdma_qpair, qpair);



	spdk_trace_record(TRACE_RDMA_QP_DISCONNECT, 0, 0, (uintptr_t)rqpair);
@@ -3562,7 +3566,7 @@ nvmf_process_cm_event(struct spdk_nvmf_transport *transport) 
			/* TODO: Should we be waiting for this event anywhere? */

			break;

		case RDMA_CM_EVENT_DISCONNECTED:

			rc = nvmf_rdma_disconnect(event);

			rc = nvmf_rdma_disconnect(event, &event_acked);

			if (rc < 0) {

				SPDK_ERRLOG("Unable to process disconnect event. rc: %d\n", rc);

				break;
@@ -3578,7 +3582,7 @@ nvmf_process_cm_event(struct spdk_nvmf_transport *transport) 
			if (event->id->qp) {

				/* If rdma_cm event has a valid `qp` pointer then the event refers to the

				 * corresponding qpair. Otherwise the event refers to a listening device. */

				rc = nvmf_rdma_disconnect(event);

				rc = nvmf_rdma_disconnect(event, &event_acked);

				if (rc < 0) {

					SPDK_ERRLOG("Unable to process disconnect event. rc: %d\n", rc);

					break;