Commit 4a745801 authored by Jim Harris's avatar Jim Harris
Browse files

bdev: add a getting started guide 



For now, this contains the documentation for configuring
block devices in configuration files.  This file can then
be used a common reference for other getting started guides -
iscsi, vhost and nvmf targets.

Signed-off-by: default avatarJim Harris <james.r.harris@intel.com>
Change-Id: Ie6f6c0b3f36dd3fdf418b904462c81a1696b9694
parent 29a3efdd

doc/Doxyfile

+5 −3
Original line number Diff line number Diff line
@@ -762,17 +762,19 @@ INPUT = ../include/spdk \ 
                         index.md \

                         directory_structure.md \

                         porting.md \

                         bdev/index.md \

                         bdev/getting_started.md \

                         event/index.md \

                         ioat/index.md \

                         iscsi/index.md \

                         iscsi/getting_started.md \

                         nvme/index.md \

                         nvme/async_completion.md \

                         nvme/fabrics.md \

                         nvme/initialization.md \

                         nvme/io_submission.md \

                         nvmf/index.md \

                         nvmf/getting_started.md \

                         iscsi/index.md \

                         iscsi/getting_started.md

                         nvmf/getting_started.md



# This tag can be used to specify the character encoding of the source files

# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses

doc/bdev/getting_started.md

0 → 100644
+89 −0
Original line number Diff line number Diff line 
# SPDK bdev Getting Started Guide {#bdev_getting_started}



Block storage in SPDK applications is provided by the SPDK bdev layer.  SPDK bdev consists of:



* a driver module API for implementing bdev drivers

* an application API for enumerating and claiming SPDK block devices and performance operations

(read, write, unmap, etc.) on those devices

* bdev drivers for NVMe, malloc (ramdisk), Linux AIO and Ceph RBD

* configuration via SPDK configuration files or JSON RPC



# Configuring block devices {#bdev_config}



SPDK block devices are typically configured via an SPDK configuration file.  These block devices

can then be associated with higher level abstractions such as iSCSI target nodes, NVMe-oF namespaces

or vhost-scsi controllers.  This section will describe how to configure block devices for the

SPDK bdev drivers included with SPDK.



## NVMe



The SPDK nvme bdev driver provides SPDK block layer access to NVMe SSDs via the SPDK userspace

NVMe driver.  The nvme bdev driver binds only to devices explicitly specified.  These devices

can be either locally attached SSDs or remote NVMe subsystems via NVMe-oF.



~~~

[Nvme]

  # NVMe Device Whitelist

  # Users may specify which NVMe devices to claim by their transport id.

  # See spdk_nvme_transport_id_parse() in spdk/nvme.h for the correct format.

  # The devices will be assigned names in the format <YourName>nY, where YourName is the

  # name specified at the end of the TransportId line and Y is the namespace id, which starts at 1.

  TransportID "trtype:PCIe traddr:0000:00:00.0" Nvme0

  TransportID "trtype:RDMA subnqn:nqn.2016-06.io.spdk:cnode1 traddr:192.168.100.1 trsvcid:4420" Nvme1

~~~



This exports block devices for all namespaces attached to the two controllers.  Block devices

for namespaces attached to the first controller will be in the format Nvme0nY, where Y is

the namespace ID.  Most NVMe SSDs have a single namespace with ID=1.  Block devices attached to

the second controller will be in the format Nvme1nY.



## Malloc



The SPDK malloc bdev driver allocates a buffer of memory in userspace as the target for block I/O

operations.  This effectively serves as a userspace ramdisk target.



Configuration file syntax:

~~~

[Malloc]

  NumberOfLuns 4

  LunSizeInMB  64

~~~



This exports 4 malloc block devices, named Malloc0 through Malloc3.  Each malloc block device will

be 64MB in size.



## Linux AIO



The SPDK aio bdev driver provides SPDK block layer access to Linux kernel block devices via Linux AIO.

Note that O_DIRECT is used and thus bypasses the Linux page cache. This mode is probably as close to

a typical kernel based target as a user space target can get without using a user-space driver.



Configuration file syntax:



~~~

[AIO]

  # normal file or block device

  AIO /dev/sdb

  AIO /dev/sdc

~~~



This exports 2 aio block devices, named AIO0 and AIO1.



## Ceph RBD



The SPDK rbd bdev driver provides SPDK block layer access to Ceph RADOS block devices (RBD).  Ceph

RBD devices are accessed via librbd and librados libraries to access the RADOS block device

exported by Ceph.



Configuration file syntax:



~~~

[Ceph]

  # The format of provided rbd info should be: Ceph rbd_pool_name rbd_name size.

  # In the following example, rbd is the name of rbd_pool; foo is the name of

  # rbd device exported by Ceph; value 512 represents the configured block size

  # for this rbd, the block size should be a multiple of 512.

  Ceph rbd foo 512

~~~



This exports 1 rbd block device, named Ceph0.

doc/bdev/index.md

0 → 100644
+3 −0
Original line number Diff line number Diff line 
# Block Device Abstraction Layer {#bdev}



- @ref bdev_getting_started

doc/index.md

+1 −0
Original line number Diff line number Diff line
@@ -24,3 +24,4 @@ which avoids kernel context switches and eliminates interrupt handling overhead. 
- @ref nvmf

- @ref ioat

- @ref iscsi

- @ref bdev

doc/iscsi/getting_started.md

+6 −93
Original line number Diff line number Diff line
@@ -21,107 +21,20 @@ TCP ports to use as iSCSI portals; general iSCSI parameters; initiator names and 
access to iSCSI target nodes; number and types of storage backends to export over iSCSI LUNs; iSCSI

target node mappings between portal groups, initiator groups, and LUNs.



The SPDK iSCSI target supports several different types of storage backends. These backends will create

SCSI LUNs which can be exported via iSCSI target nodes.

Each LUN in an iSCSI target node is associated with an SPDK block device.  See @ref bdev_getting_started

for details on configuring SPDK block devices.  The block device to LUN mappings are specified in the

configuration file as:



The storage backends can be configured in the iscsi.conf configuration file to specify the number or

size of LUNs, block device names (for exporting in-kernel block devices), or other parameters.



Currently there are 3 types of storage backends supported by the iSCSI target:



## Malloc



Configuration file syntax:

~~~

[Malloc]

  NumberOfLuns 4

  LunSizeInMB  64

~~~



Other TargetNode parameters go here (TargetName, Mapping, etc.):

~~~

~~~~

[TargetNodeX]

  LUN0 Malloc0

~~~

  LUN1 Nvme0n1

~~~~



This exports a malloc'd target. The disk is a RAM disk that is a chunk of memory allocated by iscsi in

user space. It will use offload engine to do the copy job instead of memcpy if the system has enough DMA

channels.



## Block Device



AIO devices are accessed using Linux AIO. O_DIRECT is used and thus unaligned writes will be double buffered.

This option also bypasses the Linux page cache. This mode is probably as close to a typical kernel based

target as a user space target can get without using a user-space driver.



Configuration file syntax:



~~~

[AIO]

  #normal file or block device

  AIO /dev/sdb

~~~



Other TargetNode parameters go here (TargetName, Mapping, etc.):

~~~

[TargetNodeX]

  LUN0 AIO0

~~~



## Ceph RBD



Ceph RBD devices are accessed via librbd and librados libraries to access the RADOS block device

exported by Ceph.



Configuration file syntax:



~~~

[Ceph]

  # The format of provided rbd info should be: Ceph rbd_pool_name rbd_name size.

  # In the following example, rbd is the name of rbd_pool; foo is the name of

  # rbd device exported by Ceph; value 512 represents the configured block size

  # for this rbd, the block size should be a multiple of 512.

  Ceph rbd foo 512

~~~



Other TargetNode parameters go here (TargetName, Mapping, etc.):

~~~

[TargetNodeX]

  LUN0 Ceph0

~~~



## NVMe



The SPDK NVMe driver by default binds to all NVMe controllers which are not bound to the kernel-mode

nvme driver. Users can choose to bind to fewer controllers by setting the NumControllers parameter.

Then the NVMe backend controls NVMe controller(s) directly from userspace and completely bypasses

the kernel to avoid interrupts and context switching.



~~~

[Nvme]

  # NVMe Device Whitelist

  # Users may specify which NVMe devices to claim by their transport id.

  # See spdk_nvme_transport_id_parse() in spdk/nvme.h for the correct format.

  # The second argument is the assigned name, which can be referenced from

  # other sections in the configuration file. For NVMe devices, a namespace

  # is automatically appended to each name in the format <YourName>nY, where

  # Y is the NSID (starts at 1).

  TransportID "trtype:PCIe traddr:0000:00:00.0" Nvme0

  TransportID "trtype:PCIe traddr:0000:01:00.0" Nvme1



  # The number of attempts per I/O when an I/O fails. Do not include

  # this key to get the default behavior.

  NvmeRetryCount 4



[TargetNodeX]

  # other TargetNode parameters go here (TargetName, Mapping, etc.)

  # nvme with the following format: NvmeXnY, where X = the controller ID

  # and Y = the namespace ID

  # Note: NVMe namespace IDs always start at 1, not 0 - and most

  #  controllers have only 1 namespace.

  LUN0 Nvme0n1

~~~



You should make a copy of the example configuration file, modify it to suit your environment, and

then run the iscsi_tgt application and pass it the configuration file using the -c option. Right now,

the target requires elevated privileges (root) to run.