Commit 2a5b3346 authored by Pawel Baldysiak's avatar Pawel Baldysiak Committed by Tomasz Zawadzki
Browse files

nvmf, bdev, bdev/nvme, rpc: Add support for NVMe subsystem reset (NSSR) 



Add new register in nvmf layer - to reflect NSSR support from spec.
Once NSSR is issued - pass it to all underlying namespaces (bdevs).
At the end - set CC.EN bit to 0 for all nvmf controllers
in the subsystem to disable all IO queues for all connections.
Set NSSRO bit at the end to indicate that NSSR occurs.
Add setter for csts register to have a possibility to clear
NSSRO flag.

Add parameter to rpc's create subsystem routine to control
if NSSR should be enabled or disabled for particual subsys.

Change-Id: Id96c28367f93c42230e4bf4b177c141082874e0c
Signed-off-by: default avatarPawel Baldysiak <pawel.baldysiak@dell.com>
Signed-off-by: default avatarMarcin Galecki <marcin.galecki@dell.com>
Reviewed-on: https://review.spdk.io/c/spdk/spdk/+/26001


Reviewed-by: default avatarJim Harris <jim.harris@nvidia.com>
Community-CI: Mellanox Build Bot
Reviewed-by: default avatarTomasz Zawadzki <tomasz@tzawadzki.com>
Tested-by: default avatarSPDK Automated Test System <spdkbot@gmail.com>
parent 35d33b8c

CHANGELOG.md

+6 −0
Original line number Diff line number Diff line
@@ -23,6 +23,12 @@ implementations. 
Changed the return behavior of `spdk_sock_flush`. The function now returns 0 on success, as relying

on the number of bytes returned was not recommended.



### nvmf



Add NSSR support (NVMe Subsytem Reset) to NVMe-oF target. Once NSSR is issued - it is passed to all

underlying namespaces (bdevs). Currenly only bdevs with PCIe transport would handle NSSR.

See the NVMe Subsystem Reset (NSSR) section of nvmf.md for more information.



## v25.05



### accel_mlx5

doc/jsonrpc.md

+1 −0
Original line number Diff line number Diff line
@@ -8356,6 +8356,7 @@ Construct an NVMe over Fabrics target subsystem. 
 max_discard_size_kib      | Optional   | number  | Maximum discard size (Kib). Default: 0

 max_write_zeroes_size_kib | Optional   | number  | Maximum write_zeroes size (Kib). Default: 0

 passthrough               | Optional   | boolean | Use NVMe passthrough for I/O commands and namespace-directed admin commands. Default: `false`

 enable_nssr               | Optional   | boolean | Enable NSSR (NVMe subsystem reset). Default: `false`



#### Example

doc/nvmf.md

+33 −0
Original line number Diff line number Diff line
@@ -401,3 +401,36 @@ The NVMe specification describes the method for using in-band authentication in 
establishing a secure channel (e.g. TLS).  However, that isn't supported currently, so in order to

perform in-band authentication, hosts must connect over regular listeners (i.e. those that weren't

created with the `--secure-channel` option).



## NVMe Subsystem Reset (NSSR)



NVM Subsystem Reset (NSSR) is optionally supported. When an NSSR occurs the entire NVM subsystem is

reset including Controller Level Reset (CLR) on each controller in the subsystem and disabling of

the Persistent Memory Regions (PMR) associated with those controllers.

This feature is currently limited to namespaces (bdevs) with PCIe transport.



NSSR support is disabled by default and can be enabled during creation of the subsystem.

Please see `nvmf_create_subsystem` RPC's call parameters.



During an NSSR each namespace (bdev) will additionally be reset. For bdev_nvme-backed namespaces

an NSSR will prefer to trigger an NSSR on the underlying controller.

If not supported, a CLR will be performed instead.

For other bdev types a standard bdev reset operation is performed.



NSSR resets the PCIe link and so NVMe devices appear to be hot-removed and then hot-inserted.

The same configuration steps to support PCIe hotplug of NVMe devices is required to

re-discover NVMe devices during an NSSR.



Those steps might contain:



* Unbinding the NVMe device from kernel's NVMe driver

* Binding it to vfio driver

* Attaching controller to SPDK via `bdev_nvme_attach_controller` RPC call (unless hotplug is not enabled via `bdev_nvme_set_hotplug`)

* Adding namespace to the nvmf subsystem



If bdev has multiple namespaces connected to multiple subsystems - executing NSSR on one namespace

affects the other, since the whole bdev would be removed from and added back to the system.



If a bdev is of type bdev_nvme and this underlying NVMe namespace is part of an NVM subsystem containing multiple namespaces,

then all bdevs associated with that underlying subsystem will be destroyed and re-added,

even if they are in a separate NVMe-oF subsystem that wasn't being reset. Beware of side effects.

include/spdk/bdev.h

+19 −0
Original line number Diff line number Diff line
@@ -122,6 +122,7 @@ enum spdk_bdev_io_type { 
	SPDK_BDEV_IO_TYPE_SEEK_DATA,

	SPDK_BDEV_IO_TYPE_COPY,

	SPDK_BDEV_IO_TYPE_NVME_IOV_MD,

	SPDK_BDEV_IO_TYPE_NVME_NSSR,

	SPDK_BDEV_NUM_IO_TYPES /* Keep last */

};
@@ -1950,6 +1951,24 @@ int spdk_bdev_flush_blocks(struct spdk_bdev_desc *desc, struct spdk_io_channel * 
int spdk_bdev_reset(struct spdk_bdev_desc *desc, struct spdk_io_channel *ch,

		    spdk_bdev_io_completion_cb cb, void *cb_arg);



/**

 * Submit a NVMe Subsystem Reset request to the bdev on the given channel.

 *

 * \ingroup bdev_io_submit_functions

 *

 * \param desc Block device descriptor.

 * \param ch I/O channel. Obtained by calling spdk_bdev_get_io_channel().

 * \param cb Called when the request is complete.

 * \param cb_arg Argument passed to cb.

 *

 * \return 0 on success. On success, the callback will always

 * be called (even if the request ultimately failed). Return

 * negated errno on failure, in which case the callback will not be called.

 *   * -ENOMEM - spdk_bdev_io buffer cannot be allocated

 */

int spdk_bdev_nvme_nssr(struct spdk_bdev_desc *desc, struct spdk_io_channel *ch,

			spdk_bdev_io_completion_cb cb, void *cb_arg);



/**

 * Submit abort requests to abort all I/Os which has bio_cb_arg as its callback

 * context to the bdev on the given channel.

include/spdk/nvme.h

+9 −0
Original line number Diff line number Diff line
@@ -2273,6 +2273,15 @@ int spdk_nvme_ctrlr_cmd_admin_raw(struct spdk_nvme_ctrlr *ctrlr, 
				  void *buf, uint32_t len,

				  spdk_nvme_cmd_cb cb_fn, void *cb_arg);



/**

 * Check if the controller supports NSSR

 *

 * \param ctrlr Opaque handle to NVMe controller.

 *

 * \return true if the controller supports NSSR, false otherwise

 */

bool spdk_nvme_ctrlr_is_nssr_supported(struct spdk_nvme_ctrlr *ctrlr);



/**

 * Process any outstanding completions for admin commands.

 *