markdownlint: enable rule MD025

* [vhost target](http://www.spdk.io/doc/vhost.html)

* [Virtio-SCSI driver](http://www.spdk.io/doc/virtio.html)

# In this readme

## In this readme

* [Documentation](#documentation)

* [Prerequisites](#prerequisites)

# ABI and API Deprecation {#deprecation}

# Deprecation

## ABI and API Deprecation {#deprecation}

This document details the policy for maintaining stability of SPDK ABI and API.

Specific future SPDK release for the removal must be provided.

ABI cannot be removed without providing deprecation notice for at least single SPDK release.

# Deprecation Notices {#deprecation-notices}

## Deprecation Notices {#deprecation-notices}

## bdev

### bdev

Deprecated `spdk_bdev_module_finish_done()` API, which will be removed in SPDK 22.01.

Bdev modules should use `spdk_bdev_module_fini_done()` instead.

SPDK Documentation

==================

# SPDK Documentation

The current version of the SPDK documentation can be found online at

http://www.spdk.io/doc/

Building the Documentation

==========================

## Building the Documentation

To convert the documentation into HTML run `make` in the `doc`

directory.  The output will be located in `doc/output/html`. Before

accelerated by hardware will be reported however any function can still be called, it will just be backed by

software if it is not reported as a supported capability.

# Acceleration Framework Functions {#accel_functions}

## Acceleration Framework Functions {#accel_functions}

Functions implemented via the framework can be found in the DoxyGen documentation of the

framework public header file here [accel_engine.h](https://spdk.io/doc/accel__engine_8h.html)

# Acceleration Framework Design Considerations {#accel_dc}

## Acceleration Framework Design Considerations {#accel_dc}

The general interface is defined by `/include/accel_engine.h` and implemented

in `/lib/accel`.  These functions may be called by an SPDK application and in

in hardware but if the IOAT module has been initialized and the public dualcast API

is called, it will actually be done via software behind the scenes.

# Acceleration Low Level Libraries {#accel_libs}

## Acceleration Low Level Libraries {#accel_libs}

Low level libraries provide only the most basic functions that are specific to

the hardware. Low level libraries are located in the '/lib' directory with the

The low level library for IOAT is located in `/lib/ioat`.  The low level library

for DSA is in `/liv/idxd` (IDXD stands for Intel(R) Data Acceleration Driver).

# Acceleration Plug-In Modules {#accel_modules}

## Acceleration Plug-In Modules {#accel_modules}

Plug-in modules depend on low level libraries to interact with the hardware and

add additional functionality such as queueing during busy conditions or flow

selected via startup RPC when the application is started. Otherwise, if no startup

RPC is provided, the framework is available and will use the software plug-in module.

## IOAT Module {#accel_ioat}

### IOAT Module {#accel_ioat}

To use the IOAT engine, use the RPC [`ioat_scan_accel_engine`](https://spdk.io/doc/jsonrpc.html) before starting the application.

## IDXD Module {#accel_idxd}

### IDXD Module {#accel_idxd}

To use the DSA engine, use the RPC [`idxd_scan_accel_engine`](https://spdk.io/doc/jsonrpc.html) with an optional parameter

of `-c` and provide a configuration number of either 0 or 1. These pre-defined configurations determine how the DSA engine

the module. Specialized use of DSA may require different configurations that

can be added to the module as needed.

## Software Module {#accel_sw}

### Software Module {#accel_sw}

The software module is enabled by default. If no hardware engine is explicitly

enabled via startup RPC as discussed earlier, the software module will use ISA-L

if available for functions such as CRC32C. Otherwise, standard glibc calls are

used to back the framework API.

## Batching {#batching}

### Batching {#batching}

Batching is exposed by the acceleration framework and provides an interface to

batch sets of commands up and then submit them with a single command.  The public

# Block Device User Guide {#bdev}

# Target Audience {#bdev_ug_targetaudience}

## Target Audience {#bdev_ug_targetaudience}

This user guide is intended for software developers who have knowledge of block storage, storage drivers, issuing JSON-RPC

commands and storage services such as RAID, compression, crypto, and others.

# Introduction {#bdev_ug_introduction}

## Introduction {#bdev_ug_introduction}

The SPDK block device layer, often simply called *bdev*, is a C library

intended to be equivalent to the operating system block storage layer that

provides also vbdev modules which creates block devices on existing bdev. For

example @ref bdev_ug_logical_volumes or @ref bdev_ug_gpt

# Prerequisites {#bdev_ug_prerequisites}

## Prerequisites {#bdev_ug_prerequisites}

This guide assumes that you can already build the standard SPDK distribution

on your platform. The block device layer is a C library with a single public

Detailed help for each command can be displayed by adding `-h` flag as a

command parameter.

# Configuring Block Device Modules {#bdev_ug_general_rpcs}

## Configuring Block Device Modules {#bdev_ug_general_rpcs}

Block devices can be configured using JSON RPCs. A complete list of available RPC commands

with detailed information can be found on the @ref jsonrpc_components_bdev page.

# Common Block Device Configuration Examples

## Common Block Device Configuration Examples

# Ceph RBD {#bdev_config_rbd}

## Ceph RBD {#bdev_config_rbd}

The SPDK RBD bdev driver provides SPDK block layer access to Ceph RADOS block

devices (RBD). Ceph RBD devices are accessed via librbd and librados libraries

This command will resize the Rbd0 bdev to 4096 MiB.

# Compression Virtual Bdev Module {#bdev_config_compress}

## Compression Virtual Bdev Module {#bdev_config_compress}

The compression bdev module can be configured to provide compression/decompression

services for an underlying thinly provisioned logical volume. Although the underlying

`rpc.py bdev_compress_get_orphans --name COMP_Nvme0n1`

# Crypto Virtual Bdev Module {#bdev_config_crypto}

## Crypto Virtual Bdev Module {#bdev_config_crypto}

The crypto virtual bdev module can be configured to provide at rest data encryption

for any underlying bdev. The module relies on the DPDK CryptoDev Framework to provide

`rpc.py bdev_crypto_delete CryNvmeA`

# Delay Bdev Module {#bdev_config_delay}

## Delay Bdev Module {#bdev_config_delay}

The delay vbdev module is intended to apply a predetermined additional latency on top of a lower

level bdev. This enables the simulation of the latency characteristics of a device during the functional

`rpc.py bdev_delay_delete delay0`

# GPT (GUID Partition Table) {#bdev_config_gpt}

## GPT (GUID Partition Table) {#bdev_config_gpt}

The GPT virtual bdev driver is enabled by default and does not require any configuration.

It will automatically detect @ref bdev_ug_gpt on any attached bdev and will create

possibly multiple virtual bdevs.

## SPDK GPT partition table {#bdev_ug_gpt}

### SPDK GPT partition table {#bdev_ug_gpt}

The SPDK partition type GUID is `7c5222bd-8f5d-4087-9c00-bf9843c7b58c`. Existing SPDK bdevs

can be exposed as Linux block devices via NBD and then can be partitioned with

`rpc.py nbd_stop_disk -n /dev/nbd0`

## Creating a GPT partition table using NBD {#bdev_ug_gpt_create_part}

### Creating a GPT partition table using NBD {#bdev_ug_gpt_create_part}

~~~

# Expose bdev Nvme0n1 as kernel block device /dev/nbd0 by JSON-RPC

# Nvme0n1p1 in SPDK applications.

~~~

# iSCSI bdev {#bdev_config_iscsi}

## iSCSI bdev {#bdev_config_iscsi}

The SPDK iSCSI bdev driver depends on libiscsi and hence is not enabled by default.

In order to use it, build SPDK with an extra `--with-iscsi-initiator` configure option.

The URL is in the following format:

`iscsi://[<username>[%<password>]@]<host>[:<port>]/<target-iqn>/<lun>`

# Linux AIO bdev {#bdev_config_aio}

## Linux AIO bdev {#bdev_config_aio}

The SPDK AIO bdev driver provides SPDK block layer access to Linux kernel block

devices or a file on a Linux filesystem via Linux AIO. Note that O_DIRECT is

`rpc.py bdev_aio_delete aio0`

# OCF Virtual bdev {#bdev_config_cas}

## OCF Virtual bdev {#bdev_config_cas}

OCF virtual bdev module is based on [Open CAS Framework](https://github.com/Open-CAS/ocf) - a

high performance block storage caching meta-library.

Note that OCF has a per-device RAM requirement. More details can be found in the

[OCF documentation](https://open-cas.github.io/guide_system_requirements.html).

# Malloc bdev {#bdev_config_malloc}

## Malloc bdev {#bdev_config_malloc}

Malloc bdevs are ramdisks. Because of its nature they are volatile. They are created from hugepage memory given to SPDK

application.

`rpc.py bdev_malloc_delete Malloc0`

# Null {#bdev_config_null}

## Null {#bdev_config_null}

The SPDK null bdev driver is a dummy block I/O target that discards all writes and returns undefined

data for reads.  It is useful for benchmarking the rest of the bdev I/O stack with minimal block

`rpc.py bdev_null_delete Null0`

# NVMe bdev {#bdev_config_nvme}

## NVMe bdev {#bdev_config_nvme}

There are two ways to create block device based on NVMe device in SPDK. First

way is to connect local PCIe drive and second one is to connect NVMe-oF device.

This command will remove NVMe bdev named Nvme0.

## NVMe bdev character device {#bdev_config_nvme_cuse}

### NVMe bdev character device {#bdev_config_nvme_cuse}

This feature is considered as experimental. You must configure with --with-nvme-cuse

option to enable this RPC.

`rpc.py bdev_nvme_cuse_unregister -n Nvme0`

# Logical volumes {#bdev_ug_logical_volumes}

## Logical volumes {#bdev_ug_logical_volumes}

The Logical Volumes library is a flexible storage space management system. It allows

creating and managing virtual block devices with variable size on top of other bdevs.

The SPDK Logical Volume library is built on top of @ref blob. For detailed description

please refer to @ref lvol.

## Logical volume store {#bdev_ug_lvol_store}

### Logical volume store {#bdev_ug_lvol_store}

Before creating any logical volumes (lvols), an lvol store has to be created first on

selected block device. Lvol store is lvols vessel responsible for managing underlying

`rpc.py bdev_lvol_delete_lvstore -l lvs`

## Lvols {#bdev_ug_lvols}

### Lvols {#bdev_ug_lvols}

To create lvols on existing lvol store user should use `bdev_lvol_create` RPC command.

Each created lvol will be represented by new bdev.

`rpc.py bdev_lvol_create lvol2 25 -u 330a6ab2-f468-11e7-983e-001e67edf35d`

# Passthru {#bdev_config_passthru}

## Passthru {#bdev_config_passthru}

The SPDK Passthru virtual block device module serves as an example of how to write a

virtual block device module. It implements the required functionality of a vbdev module

`rpc.py bdev_passthru_delete pt`

# Pmem {#bdev_config_pmem}

## Pmem {#bdev_config_pmem}

The SPDK pmem bdev driver uses pmemblk pool as the target for block I/O operations. For

details on Pmem memory please refer to PMDK documentation on http://pmem.io website.

`rpc.py bdev_pmem_delete pmem`

# RAID {#bdev_ug_raid}

## RAID {#bdev_ug_raid}

RAID virtual bdev module provides functionality to combine any SPDK bdevs into

one RAID bdev. Currently SPDK supports only RAID 0. RAID functionality does not

`rpc.py bdev_raid_delete Raid0`

# Split {#bdev_ug_split}

## Split {#bdev_ug_split}

The split block device module takes an underlying block device and splits it into

several smaller equal-sized virtual block devices. This serves as an example to create

`rpc.py bdev_split_delete bdev_b0`

# Uring {#bdev_ug_uring}

## Uring {#bdev_ug_uring}

The uring bdev module issues I/O to kernel block devices using the io_uring Linux kernel API. This module requires liburing.

For more information on io_uring refer to kernel [IO_uring] (https://kernel.dk/io_uring.pdf)

`rpc.py bdev_uring_delete bdev_u0`

# Virtio Block {#bdev_config_virtio_blk}

## Virtio Block {#bdev_config_virtio_blk}

The Virtio-Block driver allows creating SPDK bdevs from Virtio-Block devices.

`rpc.py bdev_virtio_detach_controller VirtioBlk0`

# Virtio SCSI {#bdev_config_virtio_scsi}

## Virtio SCSI {#bdev_config_virtio_scsi}

The Virtio-SCSI driver allows creating SPDK block devices from Virtio-SCSI LUNs.