Commit e9b5bef8 authored by Tomasz Zawadzki's avatar Tomasz Zawadzki
Browse files

doc/nvme: nvme character device documentation 



Change-Id: Ieab32f3e7aca103a270d88329d4df5fc85302795
Signed-off-by: default avatarTomasz Kulasek <tomaszx.kulasek@intel.com>
Signed-off-by: default avatarTomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/468827


Reviewed-by: default avatarJim Harris <james.r.harris@intel.com>
Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
parent ab1faf33

CHANGELOG.md

+4 −0
Original line number Diff line number Diff line
@@ -112,6 +112,10 @@ applications when a qpair is failed. This list of functions includes: 
These functions now return -ENXIO when the qpair or controller on which they

operate is failed.



EXPERIMENTAL: Added NVMe character device support to allow to create NVMe device nodes in Linux

kernel for controller as well as for namespace and process ioctl requests as usual

from linux environment.



### nvmf



The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts`

doc/bdev.md

+17 −0
Original line number Diff line number Diff line
@@ -411,6 +411,23 @@ To remove an NVMe controller use the bdev_nvme_detach_controller command. 


This command will remove NVMe bdev named Nvme0.



## NVMe bdev character device {#bdev_config_nvme_cuse}



This feature is considered as experimental.



Example commands



`rpc.py bdev_nvme_cuse_register -n Nvme0 -p spdk/nvme0`



This command will register /dev/spdk/nvme0 character device associated with Nvme0

controller. If there are namespaces created on Nvme0 controller, for each namespace

device /dev/spdk/nvme0nX is created.



Cuse devices are removed from system, when NVMe controller is detached or unregistered

with command:



`rpc.py bdev_nvme_cuse_unregister -n Nvme0`



# Logical volumes {#bdev_ug_logical_volumes}



The Logical Volumes library is a flexible storage space management system. It allows

doc/img/nvme_cuse.svg

0 → 100644
+124 −0
Original line number Diff line number Diff line 
<?xml version="1.0" encoding="UTF-8"?>

<svg width="193.94mm" height="139.71mm" version="1.1" viewBox="0 0 193.94 139.71" xmlns="http://www.w3.org/2000/svg" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">

 <title>NVMe CUSE</title>

 <defs>

  <marker id="marker9353" overflow="visible" orient="auto">

   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>

  </marker>

  <marker id="marker7156" overflow="visible" orient="auto">

   <path transform="matrix(.8 0 0 .8 10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>

  </marker>

  <marker id="marker4572" overflow="visible" orient="auto">

   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>

  </marker>

  <marker id="marker4436" overflow="visible" orient="auto">

   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>

  </marker>

  <marker id="marker4324" overflow="visible" orient="auto">

   <path transform="matrix(.8 0 0 .8 10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>

  </marker>

  <marker id="marker2300" overflow="visible" orient="auto">

   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>

  </marker>

  <marker id="marker2110" overflow="visible" orient="auto">

   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>

  </marker>

  <marker id="marker2028" overflow="visible" orient="auto">

   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>

  </marker>

  <marker id="marker1219" overflow="visible" orient="auto">

   <path transform="matrix(.8 0 0 .8 10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>

  </marker>

  <marker id="Arrow1Lstart" overflow="visible" orient="auto">

   <path transform="matrix(.8 0 0 .8 10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>

  </marker>

  <marker id="marker1127" overflow="visible" orient="auto">

   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>

  </marker>

  <marker id="Arrow1Lend" overflow="visible" orient="auto">

   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>

  </marker>

 </defs>

 <metadata>

  <rdf:RDF>

   <cc:Work rdf:about="">

    <dc:format>image/svg+xml</dc:format>

    <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>

    <dc:title>NVMe CUSE</dc:title>

   </cc:Work>

  </rdf:RDF>

 </metadata>

 <g transform="translate(-2.1066 -22.189)">

  <rect x="11.906" y="134.85" width="72.004" height="20.6" ry="3.7798" fill="none" stroke="#000" stroke-width=".5"/>

  <text x="14.363094" y="149.02231" fill="#000000" font-family="sans-serif" font-size="10.583px" letter-spacing="0px" stroke-width=".26458" word-spacing="0px" style="line-height:1.25" xml:space="preserve"><tspan x="14.363094" y="149.02231" font-family="sans-serif" font-size="3.5278px" stroke-width=".26458" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal">/dev/spdk/nvme0</tspan></text>

  <text x="47.625" y="149.02231" fill="#000000" font-family="sans-serif" font-size="10.583px" letter-spacing="0px" stroke-width=".26458" word-spacing="0px" style="line-height:1.25" xml:space="preserve"><tspan x="47.625" y="149.02231" font-family="sans-serif" font-size="3.5278px" stroke-width=".26458" writing-mode="lr" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal">/dev/spdk/nvme0n1</tspan></text>

  <g stroke="#000">

   <rect x="12.095" y="35.818" width="71.249" height="88.446" ry="4.3467" fill="none" stroke-width=".5"/>

   <rect x="133.43" y="33.929" width="62.366" height="76.351" ry="4.7247" fill="none" stroke-width=".5"/>

   <g fill="#fff" stroke-width=".26458">

    <rect x="14.174" y="91.57" width="64.256" height="24.568"/>

    <g fill-opacity=".9798">

     <rect x="46.302" y="100.64" width="26.62" height="11.061"/>

    </g>

   </g>

   <g transform="translate(-.53932 -.16291)">

    <path d="m63.878 111.98v32.884" fill="none" marker-end="url(#marker1127)" marker-start="url(#Arrow1Lstart)" stroke-width=".26458px"/>

    <g stroke-width=".265">

     <path d="m34.585 115.57v28.726" fill="none" marker-end="url(#Arrow1Lend)" marker-start="url(#marker1219)"/>

     <rect x="136.26" y="39.031" width="54.996" height="58.586" fill="#fff"/>

     <rect x="153.84" y="52.26" width="34.018" height="11.906" ry="5.8544" fill="none"/>

    </g>

    <path d="m112.45 24.479v137.58" fill="none" stroke-dasharray="1.5874999, 1.5874999" stroke-width=".26458"/>

   </g>

   <g fill="#fff" stroke-width=".265">

    <rect x="89.58" y="54.339" width="38.365" height="8.8824"/>

   </g>

  </g>

  <g font-family="sans-serif" font-size="4.2333px" letter-spacing="0px" stroke-width=".26458" word-spacing="0px">

   <text x="93.54911" y="59.800339" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="93.54911" y="59.800339" stroke-width=".26458">io_msg queue</tspan></text>

   <text x="11.906249" y="27.31399" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="11.906249" y="27.31399" stroke-width=".26458">CUSE threads</tspan></text>

   <text x="165.36458" y="27.502975" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="165.36458" y="27.502975" stroke-width=".26458">SPDK threads</tspan></text>

  </g>

  <g stroke="#000">

   <rect x="17.009" y="47.914" width="29.482" height="13.04" ry="6.5201" fill="#fff" stroke-width=".265"/>

   <rect x="49.921" y="68.161" width="28.915" height="13.04" ry="6.5201" fill="#fff" stroke-width=".265"/>

   <g fill="none">

    <path d="m32.506 61.143v30.427" marker-start="url(#marker7156)" stroke-width=".26458px"/>

    <path d="m63.689 81.176 0.18899 19.277" marker-start="url(#marker4324)" stroke-width=".265"/>

    <g stroke-width=".26458px">

     <path d="m46.113 54.339h43.467" marker-end="url(#marker2028)"/>

     <path d="m64.284 67.972c0.02768-6.3997-1.3229-5.2917 25.135-5.2917" marker-end="url(#marker2110)"/>

     <path d="m127.78 56.066h25.135" marker-end="url(#marker2300)"/>

    </g>

   </g>

  </g>

  <g stroke-width=".26458">

   <g transform="translate(-.25341)" font-family="sans-serif" font-size="4.2333px" letter-spacing="0px" word-spacing="0px">

    <text x="138.90625" y="44.889877" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="138.90625" y="44.889877" stroke-width=".26458">NVMe</tspan></text>

    <text x="16.063986" y="97.050598" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="16.063986" y="97.050598" stroke-width=".26458">CUSE ctrlr</tspan></text>

    <text x="48.380947" y="106.12202" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="48.380947" y="106.12202" stroke-width=".26458">CUSE ns</tspan></text>

    <text x="51.420551" y="75.799461" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="51.420551" y="75.799461" stroke-width=".26458">ioctl pthread</tspan></text>

    <text x="18.906757" y="55.833015" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="18.906757" y="55.833015" stroke-width=".26458">ioctl pthread</tspan></text>

   </g>

   <path d="m160.86 85.17c0.38097 13.154-7.1538 11.542-82.052 10.936" fill="none" marker-end="url(#marker4572)" stroke="#000" stroke-dasharray="0.79374995, 0.79374995"/>

   <path d="m179.38 85.17c0.37797 22.25-6.5765 20.83-106.08 20.641" fill="none" marker-end="url(#marker4436)" stroke="#000" stroke-dasharray="0.79374995, 0.79374995"/>

  </g>

  <g font-family="sans-serif" font-size="4.2333px" letter-spacing="0px" stroke-width=".26458" word-spacing="0px">

   <text x="13.229166" y="139.7619" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="13.229166" y="139.7619" stroke-width=".26458">Kernel</tspan></text>

   <text x="14.552083" y="41.488094" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="14.552083" y="41.488094" stroke-width=".26458">CUSE</tspan></text>

   <text x="161.73709" y="59.415913" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="161.73709" y="59.415913" stroke-width=".26458">io poller</tspan></text>

  </g>

  <g fill="none" stroke="#000">

   <path d="m111.91 127.5h-109.8" stroke-dasharray="1.58749992, 1.58749992" stroke-width=".26458"/>

   <rect x="153.3" y="71.941" width="34.018" height="13.229" ry="6.6146" stroke-width=".265"/>

   <path d="m170.12 64.003v7.9375" marker-end="url(#marker9353)" stroke-width=".265"/>

  </g>

  <g font-family="sans-serif" font-size="4.2333px" letter-spacing="0px" stroke-width=".26458" word-spacing="0px">

   <text x="159.72221" y="79.76664" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="159.72221" y="79.76664" stroke-width=".26458">io execute</tspan></text>

   <text x="172.34003" y="68.59539" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="172.34003" y="68.59539" font-family="sans-serif" font-size="2.8222px" stroke-width=".26458" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal">fn(arg)</tspan></text>

   <text x="53.046707" y="52.192699" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="53.046707" y="52.192699" font-family="sans-serif" font-size="2.8222px" stroke-width=".26458" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal">nvme_io_msg send()</tspan></text>

   <text x="53.102341" y="60.250244" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="53.102341" y="60.250244" font-family="sans-serif" font-size="2.8222px" stroke-width=".26458" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal">nvme_io_msg send()</tspan></text>

   <text x="120.79763" y="50.70586" font-size="12px" stroke-width="1" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="120.79763" y="50.70586" font-family="sans-serif" font-size="2.8222px" stroke-width=".26458" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal">spdk_nvme_io_msg process()</tspan></text>

  </g>

 </g>

</svg>

doc/jsonrpc.md

+74 −0
Original line number Diff line number Diff line
@@ -1627,6 +1627,80 @@ Example response: 
}

~~~



## bdev_nvme_cuse_register {#rpc_bdev_nvme_cuse_register}



Register CUSE device on NVMe controller.

This feature is considered as experimental.



### Parameters



Name                    | Optional | Type        | Description

----------------------- | -------- | ----------- | -----------

name                    | Required | string      | Name of the NVMe controller

dev_path                | Required | string      | Path to the CUSE controller device, e.g. spdk/nvme0



### Example



Example request:



~~~

{

  "params": {

    "dev_path": "spdk/nvme0",

    "name": "Nvme0"

  },

  "jsonrpc": "2.0",

  "method": "bdev_nvme_cuse_register",

  "id": 1

}

~~~



Example response:



~~~

{

  "jsonrpc": "2.0",

  "id": 1,

  "result": true

}

~~~



## bdev_nvme_cuse_unregister {#rpc_bdev_nvme_cuse_unregister}



Unregister CUSE device on NVMe controller.

This feature is considered as experimental.



### Parameters



Name                    | Optional | Type        | Description

----------------------- | -------- | ----------- | -----------

name                    | Required | string      | Name of the NVMe controller



### Example



Example request:



~~~

{

  "params": {

    "name": "Nvme0"

  },

  "jsonrpc": "2.0",

  "method": "bdev_nvme_cuse_unregister",

  "id": 1

}

~~~



Example response:



~~~

{

  "jsonrpc": "2.0",

  "id": 1,

  "result": true

}

~~~



## bdev_rbd_create {#rpc_bdev_rbd_create}



Create @ref bdev_config_rbd bdev

doc/nvme.md

+29 −0
Original line number Diff line number Diff line
@@ -9,6 +9,7 @@ 
* @ref nvme_fabrics_host

* @ref nvme_multi_process

* @ref nvme_hotplug

* @ref nvme_cuse



# Introduction {#nvme_intro}
@@ -266,3 +267,31 @@ This means I/O in flight during a hot remove will complete with an appropriate e 
code and will not crash the application.



@sa spdk_nvme_probe



# NVMe Character Devices {#nvme_cuse}



This feature is considered as experimental.



![NVMe character devices processing diagram](nvme_cuse.svg)



For each controller as well as namespace, character devices are created in the

locations:

~~~{.sh}

    /dev/'dev_path'

    /dev/'dev_path'nY

    ...

~~~



Requests from CUSE are handled by pthreads when controller and namespaces are created.

Those pass the I/O or admin commands via a ring to a thread that processes them using

spdk_nvme_io_msg_process().



Ioctls that request information attained when attaching NVMe controller receive an

immediate response, without passing them through the ring.



This interface reserves one qpair for sending down the I/O for each controller.



## Enabling cuse support for NVMe



Cuse support is disabled by default. To enable support for NVMe devices SPDK

must be compiled with "./configure --with-nvme-cuse".