docs: Add documentation for peer-2-peer

                         nvme-cli.md \

                         nvmf.md \

                         nvmf_tgt_pg.md \

                         peer_2_peer.md \

                         ssd_internals.md \

                         userspace.md \

                         vagrant.md \

- @ref event

- @ref blob

# Miscellaneous {#misc}

- @ref peer_2_peer

# Modules {#modules}

- @ref nvme

# Peer-2-Peer DMAs {#peer_2_peer}

Please note that the functionality discussed in this document is

currently tagged as experimental.

# In this document {#p2p_toc}

* @ref p2p_overview

* @ref p2p_nvme_api

* @ref p2p_cmb_copy

* @ref p2p_issues

# Overview {#p2p_overview}

Peer-2-Peer (P2P) is the concept of DMAing data directly from one PCI

End Point (EP) to another without using a system memory buffer. The

most obvious example of this from an SPDK perspective is using a NVMe

Controller Memory Buffer (CMB) to enable direct copies of data between

two NVMe SSDs.

In this section of documentation we outline how to perform P2P

operations in SPDK and outline some of the issues that can occur when

performing P2P operations.

# The P2P API for NVMe {#p2p_nvme_api}

The functions that provide access to the NVMe CMBs for P2P

capabilities are given in the table below.

Key Functions                               | Description

------------------------------------------- | -----------

spdk_nvme_ctrlr_alloc_cmb_io_buffer()       | @copybrief spdk_nvme_ctrlr_alloc_cmb_io_buffer()

spdk_nvme_ctrlr_free_cmb_io_buffer()        | @copybrief spdk_nvme_ctrlr_free_cmb_io_buffer()

# cmb_copy: An example P2P Application {#p2p_cmb_copy}

Run the cmb_copy example application.

~~~{.sh}

./examples/nvme/cmb_copy -r <pci id of write ssd>-1-0-1 -w <pci id of write ssd>-1-0-1 -c <pci id of the ssd with cmb>

~~~

This should copy a single LBA (LBA 0) from namespace 1 on the read

NVMe SSD to LBA 0 on namespace 1 on the write SSD using the CMB as the

DMA buffer.

# Issues with P2P {#p2p_issues}

* In some systems when performing peer-2-peer DMAs between PCIe EPs

  that are directly connected to the Root Complex (RC) the DMA may

  fail or the performance may not be great. Basically your milage may

  vary. It is recommended that you use a PCIe switch (such as those

  provided by Broadcom or Microsemi) as that is know to provide good

  performance.

* Even with a PCIe switch there may be occasions where peer-2-peer

  DMAs fail to work. This is probaby due to PCIe Access Control

  Services (ACS) being enabled by the BIOS and/or OS. You can disable

  ACS using setpci or via out of tree kernel patches that can be found

  on the internet.

* In more complex topologies involving several switches it may be

  possible to construct multiple paths between EPs. This could lead to

  TLP ordering problems. If you are working in these environments be

  careful!

				    struct spdk_nvme_status *completion_status);

/**

 * \brief Allocate an I/O buffer from the controller memory buffer.

 * \brief Allocate an I/O buffer from the controller memory buffer (Experimental).

 *

 * \param ctrlr Controller from which to allocate memory buffer.

 * \param size Size of buffer to allocate in bytes.

 *

 * \return Pointer to controller memory buffer allocation, or NULL if allocation was not possible.

 *

 * This function allocates registered memory which belongs to the

 * Controller Memory Buffer (CMB) of the specified NVMe

 * controller. Note that the CMB has to support the WDS and RDS

 * capabilities for the allocation to be successful. Also, due to

 * vtophys contraints the CMB must be at least 4MiB in size. Free

 * memory allocated with this function using

 * spdk_nvme_ctrlr_free_cmb_io_buffer().

 */

void *spdk_nvme_ctrlr_alloc_cmb_io_buffer(struct spdk_nvme_ctrlr *ctrlr, size_t size);

/**

 * \brief Free a controller memory I/O buffer.

 * \brief Free a controller memory I/O buffer (Experimental).

 *

 * \param ctrlr Controller from which the buffer was allocated.

 * \param buf Buffer previously allocated by spdk_nvme_ctrlr_alloc_cmb_io_buffer().

 * \param size Size of buf in bytes.

 *

 * Note this function is currently a NOP which is not a good thing and

 * is one reason why this and spdk_nvme_ctrlr_alloc_cmb_io_buffer()

 * are currently marked as experimental.

 */

void spdk_nvme_ctrlr_free_cmb_io_buffer(struct spdk_nvme_ctrlr *ctrlr, void *buf, size_t size);