Commit c2735134 authored by Denis Nagorny's avatar Denis Nagorny Committed by Tomasz Zawadzki
Browse files

nvme/rdma: Allows to use PCI Express Relaxed Ordering 



This fix allows to use relaxed ordering feature where it is
supported. libibversb checks with the driver if relaxed ordering
access flag is supported and ignores it if not.

Experiments show that set by default it doesn't spoil performance but
allows to reach desired one on AMD EPYC systems. For example fio read
test (ConnectX-6, AMD EPYC 7763, two jobs, queue depth 32, block size
32K) can starve down to 6-7 GiB/s without it. Enabling this option
allows to get bandwidth more than 21 GiB/s.

Change-Id: I5983aed5d1f38ee7bec9c310597731c9a6a329da
Signed-off-by: default avatarDenis Nagorny <denisn@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14885


Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: default avatarJim Harris <james.r.harris@intel.com>
Reviewed-by: default avatarBen Walker <benjamin.walker@intel.com>
parent aba9c09c

lib/rdma/common.c

+3 −0
Original line number Diff line number Diff line
@@ -68,6 +68,9 @@ rdma_mem_notify(void *cb_ctx, struct spdk_mem_map *map, 
			default:

				SPDK_UNREACHABLE();

			}

#ifdef IBV_ACCESS_OPTIONAL_FIRST

			access_flags |= IBV_ACCESS_RELAXED_ORDERING;

#endif

			mr = ibv_reg_mr(pd, vaddr, size, access_flags);

			if (mr == NULL) {

				SPDK_ERRLOG("ibv_reg_mr() failed\n");