Commit 2e283fcb authored by 0xe0f's avatar 0xe0f Committed by Tomasz Zawadzki
Browse files

bdev/daos: introduction of daos bdev 

This commmit introduces a new bdev type backed up by DAOS DFS.

Design wise this bdev is a file named as the bdev itself in the DAOS POSIX
container that uses daos event queue per io channel.
Having an event queue per io channel is showing the best IO throughput.
The implementation uses the independent pool and container connections per
device's channel for the best IO throughput.

The semantic of usage is the same as any other bdev type.

To build SPDK with daos support, daos-devel package has to be installed.
The current supported DAOS version is v2.X, please see the installatoin and
setup guide here: https://docs.daos.io/v2.0/

$ ./configure --with-daos

To run it, the target machine should have daos_agent up and running, as
well as the pool and POSIX container ready to use, please see the
detailed requirements here: https://docs.daos.io/v2.0/admin/hardware/

.

To export bdev over tcp:

$ ./nvmf_tgt &
$ ./scripts/rpc.py nvmf_create_transport -t TCP -u 2097152 -i 2097152

$ ./scripts/rpc.py bdev_daos_create daosdev0 <pool-label> <cont-label>
1048576 4096

$ ./scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk1:cnode1 -a -s
SPDK00000000000001 -d SPDK_Virtual_Controller_1
$ ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk1:cnode1
daosdev0
$ ./scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk1:cnode1
-t tcp -a <IP> -s 4420

On the initiator side, make sure that `nvme-tcp` module is loaded then
connect drives, for instance:

$ nvme connect-all -t tcp -a 172.31.91.61 -s 4420
$ nvme list

Signed-off-by: default avatarDenis Barakhtanov <denis.barahtanov@croit.io>
Change-Id: I51945465122e0fb96de4326db742169419966806
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12260


Community-CI: Mellanox Build Bot
Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: default avatarJim Harris <james.r.harris@intel.com>
Reviewed-by: default avatarBen Walker <benjamin.walker@intel.com>
parent 711759a0

CONFIG

+4 −0
Original line number Diff line number Diff line
@@ -104,6 +104,10 @@ CONFIG_FC_PATH= 
# Requires librbd development libraries

CONFIG_RBD=n



# Build DAOS support in bdev modules

# Requires daos development libraries

CONFIG_DAOS=n



# Build vhost library.

CONFIG_VHOST=y

configure

+18 −0
Original line number Diff line number Diff line
@@ -78,6 +78,8 @@ function usage() { 
	echo " --without-fc              If an argument is provided, it is considered a directory containing"

	echo "                           libufc.a and fc_lld.h. Otherwise the regular system paths will"

	echo "                           be searched."

	echo " --with-daos               Build DAOS bdev module."

	echo " --without-daos            No path required."

	echo " --with-shared             Build spdk shared libraries."

	echo " --without-shared          No path required."

	echo " --with-iscsi-initiator    Build with iscsi bdev module."
@@ -483,6 +485,12 @@ for i in "$@"; do 
			CONFIG[FC]=n

			CONFIG[FC_PATH]=

			;;

		--with-daos)

			CONFIG[DAOS]=y

			;;

		--without-daos)

			CONFIG[DAOS]=n

			;;

		--with-shared)

			CONFIG[SHARED]=y

			;;
@@ -1036,6 +1044,16 @@ if [[ "${CONFIG[ISCSI_INITIATOR]}" = "y" ]]; then 
	fi

fi



if [[ "${CONFIG[DAOS]}" = "y" ]]; then

	if ! echo -e '#include <daos.h>\n#include <daos_fs.h>\n' \

		'int main(void) { return 0; }\n' \

		| "${BUILD_CMD[@]}" -lgurt -ldaos -ldaos_common -ldfs - 2> /dev/null; then

		echo "--with-daos requires libdaos, libdaos_common, libdfs and libgurt"

		echo "Please install then re-run this script."

		exit 1

	fi

fi



if [[ "${CONFIG[ASAN]}" = "y" ]]; then

	if ! echo -e 'int main(void) { return 0; }\n' \

		| "${BUILD_CMD[@]}" -fsanitize=address - 2> /dev/null; then

doc/bdev.md

+21 −0
Original line number Diff line number Diff line
@@ -672,3 +672,24 @@ Virtio-SCSI devices can be removed with the following command 
`rpc.py bdev_virtio_detach_controller VirtioScsi0`



Removing a Virtio-SCSI device will destroy all its bdevs.



## DAOS bdev {#bdev_config_daos}



DAOS bdev creates SPDK block device on top of DAOS DFS, the name of the bdev defines the file name in DFS namespace.

Note that DAOS container has to be POSIX type, e.g.: ` daos cont create --pool=test-pool --label=test-cont --type=POSIX`



To build SPDK with daos support, daos-devel package has to be installed, please see the setup [guide](https://docs.daos.io/v2.0/).

To enable the module, configure SPDK using `--with-daos` flag.



Running `daos_agent` service on the target machine is required for the SPDK DAOS bdev communication with a DAOS cluster.



The implementation uses the independent pool and container connections per device's channel for the best IO throughput, therefore,

running a target application with multiple cores (`-m [0-7], for example) is highly advisable.



Example command for creating daos bdev:



`rpc.py bdev_daos_create daosdev0 test-pool test-cont 64 4096`



Example command for removing daos bdev:



`rpc.py bdev_daos_delete daosdev0`

doc/jsonrpc.md

+86 −1
Original line number Diff line number Diff line
@@ -481,7 +481,9 @@ Example response: 
    "bdev_lvol_create",

    "bdev_lvol_delete_lvstore",

    "bdev_lvol_rename_lvstore",

    "bdev_lvol_create_lvstore"

    "bdev_lvol_create_lvstore",

    "bdev_daos_delete",

    "bdev_daos_create"

  ]

}

~~~
@@ -10141,3 +10143,86 @@ Example response: 
}



~~~



### bdev_daos_create {#rpc_bdev_daos_create}



Construct @ref bdev_config_daos



#### Parameters



Name                    | Optional | Type        | Description

----------------------- | -------- | ----------- | -----------

name                    | Required | string      | Bdev name to use

pool                    | Required | string      | DAOS pool label or its uuid

cont                    | Required | string      | DAOS cont label or its uuid

block_size              | Required | number      | Block size in bytes -must be multiple of 512

num_blocks              | Required | number      | Number of blocks

uuid                    | Optional | string      | UUID of new bdev



#### Result



Name of newly created bdev.



#### Example



Example request:



~~~json

{

  "params": {

    "block_size": 4096,

    "num_blocks": 16384,

    "name": "daosdev0",

    "pool": "test-pool",

    "cont": "test-cont",

  },

  "jsonrpc": "2.0",

  "method": "bdev_daos_create",

  "id": 1

}

~~~



Example response:



~~~json

{

  "jsonrpc": "2.0",

  "id": 1,

  "result": "daosdev0"

}

~~~



### bdev_daos_delete {#rpc_bdev_daos_delete}



Delete @ref bdev_config_daos



#### Parameters



Name                    | Optional | Type        | Description

----------------------- | -------- | ----------- | -----------

name                    | Required | string      | Bdev name



#### Example



Example request:



~~~json

{

  "params": {

    "name": "daosdev0"

  },

  "jsonrpc": "2.0",

  "method": "bdev_daos_delete",

  "id": 1

}

~~~



Example response:



~~~json

{

  "jsonrpc": "2.0",

  "id": 1,

  "result": true

}

~~~

mk/spdk.modules.mk

+5 −0
Original line number Diff line number Diff line
@@ -84,6 +84,11 @@ BLOCKDEV_MODULES_LIST += bdev_pmem 
BLOCKDEV_MODULES_PRIVATE_LIBS += -lpmemblk -lpmem

endif



ifeq ($(CONFIG_DAOS),y)

BLOCKDEV_MODULES_LIST += bdev_daos

BLOCKDEV_MODULES_PRIVATE_LIBS += -ldaos -ldaos_common -ldfs -lgurt -luuid -ldl

endif



SOCK_MODULES_LIST = sock_posix



ifeq ($(OS), Linux)