Commit 6a59daad authored by Shuhei Matsumoto's avatar Shuhei Matsumoto Committed by Tomasz Zawadzki
Browse files

nvme_rdma: Poll disconnect until completion if async mode is disabled 



nvme_rdma_ctrlr_disconnect_qpair() does not poll the qpair until it is
actually disconnected if it is in a poll group even if its async mode
is disabled. Hence, spdk_nvme_ctrlr_free_io_qpair() removes the qpair
from a poll group when it is being disconnected.

On the other hand, I/O qpair is destroyed after it is actually
disconnected.

When SRQ is enabled and used, a SRQ is destroyed if the corresponding
poller does not have any I/O qpair after an I/O qpair is removed from
the poller.

In particular, if we use spdk_nvme_ctrlr_free_io_qpair(), a SRQ is
destroyed before the corresponding I/O qpairs are destroyed.
Destroying a SRQ failed because it is still referenced by I/O qpairs.

This bug was found when running the SPDK NVMe perf tool with SRQ.

The reason was we had nvme_rdma_poll_group_process_completions() to call
disconnected_qpair_cb after the qpair is actually disconnected.
However, it is ensured that nvme_rdma_poll_group_process_completions()
calls disconnected_qpair_cb for any disconnected qpair.

Hence, remove a check if qpair->poll_group is not NULL from
nvme_rdma_ctrlr_disconnect_qpair() and update the comment.

Signed-off-by: default avatarShuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I0fde0d827eec3280e1cc5a0fce34d163a6069bc4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14908


Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: default avatarBen Walker <benjamin.walker@intel.com>
Reviewed-by: default avatarAleksey Marchuk <alexeymar@nvidia.com>
parent 3fcee8dd

lib/nvme/nvme_rdma.c

+5 −7
Original line number Diff line number Diff line
@@ -2018,14 +2018,12 @@ nvme_rdma_ctrlr_disconnect_qpair(struct spdk_nvme_ctrlr *ctrlr, struct spdk_nvme 


	_nvme_rdma_ctrlr_disconnect_qpair(ctrlr, qpair, nvme_rdma_qpair_disconnected);



	/* If the qpair is in a poll group, disconnected_qpair_cb has to be called

	 * asynchronously after the qpair is actually disconnected. Hence let

	 * poll_group_process_completions() poll the qpair until then.

	 *

	 * If the qpair is not in a poll group, poll the qpair until it is actually

	 * disconnected here.

	/* If the async mode is disabled, poll the qpair until it is actually disconnected.

	 * It is ensured that poll_group_process_completions() calls disconnected_qpair_cb

	 * for any disconnected qpair. Hence, we do not have to check if the qpair is in

	 * a poll group or not.

	 */

	if (qpair->async || qpair->poll_group != NULL) {

	if (qpair->async) {

		return;

	}