doc: describe dynamic memory management

callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc

in lieu of the notify callback.

### DPDK 18.08

The DPDK submodule has been updated to the DPDK 18.08 release. SPDK will now automatically

utilize DPDK's dynamic memory management with DPDK versions >= 18.05.1.

Hugepages can be still reserved with `[-s|--mem-size <size>]` option at application startup,

but once we use them all up, instead of failing user allocations with -ENOMEM, we'll try

to dynamically reserve even more. This allows starting SPDK with `--mem-size 0` and using

only as many hugepages as it is really needed.

Due to this change, the memory buffers returned by `spdk_*malloc()` are no longer guaranteed

to be physically contiguous.

### iscsi target

Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC

the highest page size. This option accepts a number of bytes with a possible

binary prefix, e.g. 1024, 1024M, 1G. The default unit is megabyte.

Starting with DPDK 18.05.1, it's possible to reserve hugepages at runtime, meaning

that SPDK application can be started with 0 pre-reserved memory. Unlike hugepages

pre-reserved at the application startup, the hugepages reserved at runtime will be

released to the system as soon as they're no longer used.

### Disable PCI access {#cmd_arg_disable_pci_access}

If SPDK is run with PCI access disabled it won't detect any PCI devices. This

differently than regular 4KiB pages. Specifically, the operating system will

never change their physical location. This is not by intent, and so things

could change in future versions, but it is true today and has been for a number

of years (see the later section on the IOMMU for a future-proof solution). DPDK

goes through great pains to allocate hugepages such that it can string together

the longest runs of physical pages possible, such that it can accommodate

physically contiguous allocations larger than a single page.

of years (see the later section on the IOMMU for a future-proof solution).

With this explanation, hopefully it is now clear why all data buffers passed to

SPDK must be allocated using spdk_dma_malloc() or its siblings. The buffers

for just that subsystem; this allows the user to skip the discovery step

and connect directly to a subsystem with a known address.

## RDMA Limitations

Please refer to NVMe-oF target's @ref nvmf_rdma_limitations

# NVMe Multi Process {#nvme_multi_process}

This capability enables the SPDK NVMe driver to support multiple processes accessing the

SPDK has a tracing framework for capturing low-level event information at runtime.

@ref nvmf_tgt_tracepoints enable analysis of both performance and application crashes.

## RDMA Limitations {#nvmf_rdma_limitations}

As RDMA NICs put a limitation on the number of memory regions registered, the SPDK NVMe-oF

target application may eventually start failing to allocate more DMA-able memory. This is

an imperfection of the DPDK dynamic memory management and is most likely to occur with too

many 2MB hugepages reserved at runtime. Some of our NICs report as many as 2048 for the

maximum number of memory regions, meaning that exactly that many pages can be allocated.

With 2MB hugepages, this gives us a 4GB memory limit. It can be overcome by using 1GB

hugepages or by pre-reserving memory at application startup with `--mem-size` or `-s`

option. All pre-reserved memory will be registered as a single region, but won't be

returned to the system until the SPDK application is terminated.