Commit f89cf818 authored Apr 06, 2022 by Alex Michon Committed by Tomasz Zawadzki Apr 27, 2022

nvme/pcie: Fix doorbell delay with fuse operations



When sending the first part of a fuse command, we set the
first_fused_submitted flag so that we don't ring the doorbell
immediately. When the second part is sent, we ring the doorbell for
both commands.
However, this doesn't work well when we use the option to delay ringing
the doorbell. We send both parts, then later when we try to ring the
doorbell, we don't because of the first_fused_submitted flag from the
first command.
Replace this mechanism by keeping track of the last submitted fuse.

Change-Id: Ia4ac9b3ce9c319ee4c7e42f86eadda93dac85fca
Signed-off-by: Alex Michon <amichon@kalrayinc.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12182


Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>

parent 6e2ba81e

lib/nvme/nvme_internal.h

+1 −1

Original line number	Diff line number	Diff line
		@@ -446,7 +446,7 @@ struct spdk_nvme_qpair {
		*/
		uint8_t no_deletion_notification_needed: 1;

		uint8_t first_fused_submitted: 1;
		uint8_t last_fuse: 2;

		uint8_t transport_failure_reason: 2;
		uint8_t last_transport_failure_reason: 2;

lib/nvme/nvme_pcie_common.c

+6 −3

Original line number	Diff line number	Diff line
		@@ -651,9 +651,12 @@ nvme_pcie_qpair_submit_tracker(struct spdk_nvme_qpair *qpair, struct nvme_tracke
		spdk_trace_record(TRACE_NVME_PCIE_SUBMIT, qpair->id, 0, (uintptr_t)req,
		req->cmd.cid, req->cmd.opc, req->cmd.cdw10, req->cmd.cdw11, req->cmd.cdw12);

		if (req->cmd.fuse == SPDK_NVME_IO_FLAGS_FUSE_FIRST) {
		/* This is first cmd of two fused commands - don't ring doorbell */
		qpair->first_fused_submitted = 1;
		if (req->cmd.fuse) {
		/*
		* Keep track of the fuse operation sequence so that we ring the doorbell only
		* after the second fuse is submitted.
		*/
		qpair->last_fuse = req->cmd.fuse;
		}

		/* Don't use wide instructions to copy NVMe command, this is limited by QEMU

lib/nvme/nvme_pcie_internal.h

+1 −2

Original line number	Diff line number	Diff line
		@@ -277,9 +277,8 @@ nvme_pcie_qpair_ring_sq_doorbell(struct spdk_nvme_qpair *qpair)
		struct nvme_pcie_ctrlr *pctrlr = nvme_pcie_ctrlr(qpair->ctrlr);
		bool need_mmio = true;

		if (qpair->first_fused_submitted) {
		if (qpair->last_fuse == SPDK_NVME_IO_FLAGS_FUSE_FIRST) {
		/* This is first cmd of two fused commands - don't ring doorbell */
		qpair->first_fused_submitted = 0;
		return;
		}