Commit 6e7c78dd authored by Daniel Verkamp's avatar Daniel Verkamp Committed by Jim Harris
Browse files

doc/blobfs: clean up RocksDB guide formatting 



Make the shell fragments highlighted as code blocks and fix up a few
minor typos.

Change-Id: If9992df414fe85c6173d440061736092e8d609ba
Signed-off-by: default avatarDaniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/379908


Tested-by: default avatarSPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: default avatarJim Harris <james.r.harris@intel.com>
parent c94bb669

doc/blobfs.md

+49 −28
Original line number Diff line number Diff line
@@ -4,52 +4,71 @@ 


# RocksDB Integration {#blobfs_rocksdb}



1. Clone and build the SPDK repository as per https://github.com/spdk/spdk

Clone and build the SPDK repository as per https://github.com/spdk/spdk



~~~{.sh}

git clone https://github.com/spdk/spdk.git

cd spdk

./configure

make

~~~



2. Into a separate directory, clone the RocksDB git repo from the SPDK GitHub fork.  Make sure you check out the spdk branch.

Clone the RocksDB repository from the SPDK GitHub fork into a separate directory.

Make sure you check out the `spdk-v5.6.1` branch.



~~~{.sh}

cd ..

git clone -b spdk-v5.6.1 https://github.com/spdk/rocksdb.git

~~~



3. Build RocksDB.  Only the db_bench benchmarking tool is integrated with BlobFS.

   (Note: add "DEBUG_LEVEL=0" for a release build.)

Build RocksDB.  Only the `db_bench` benchmarking tool is integrated with BlobFS.

(Note: add `DEBUG_LEVEL=0` for a release build.)



~~~{.sh}

cd rocksdb

make db_bench SPDK_DIR=path/to/spdk

~~~



4. Copy etc/spdk/rocksdb.conf.in from the spdk repository to /usr/local/etc/spdk/rocksdb.conf.

Copy `etc/spdk/rocksdb.conf.in` from the SPDK repository to `/usr/local/etc/spdk/rocksdb.conf`.



~~~{.sh}

cd ../spdk

cp etc/spdk/rocksdb.conf.in /usr/local/etc/spdk/rocksdb.conf

~~~



5. Append an NVMe section to the configuration file using SPDK's gen_nvme.sh script.

Append an NVMe section to the configuration file using SPDK's `gen_nvme.sh` script.



~~~{.sh}

scripts/gen_nvme.sh >> /usr/local/etc/spdk/rocksdb.conf

~~~



6. Verify the configuration file has specified the correct NVMe SSD.  If there are any NVMe SSDs you do not wish to use for RocksDB/SPDK testing, remove them from the configuration file.

Verify the configuration file has specified the correct NVMe SSD.

If there are any NVMe SSDs you do not wish to use for RocksDB/SPDK testing, remove them from the configuration file.



7. Make sure you have at least 5GB of memory allocated for huge pages.  By default the SPDK setup.sh script only allocates 2GB (1024 huge pages).  The following will allocate 5GB worth of 2MB huge pages (in addition to binding the NVMe devices to uio/vfio).  If using 1GB huge pages, adjust the NRHUGE value accordingly.

Make sure you have at least 5GB of memory allocated for huge pages.

By default, the SPDK `setup.sh` script only allocates 2GB (1024 huge pages).

The following will allocate 5GB worth of 2MB huge pages (in addition to binding the NVMe devices to uio/vfio).

If using 1GB huge pages, adjust the `NRHUGE` value accordingly.



~~~{.sh}

NRHUGE=2560 scripts/setup.sh

~~~



8. Create an empty SPDK blobfs for testing.

Create an empty SPDK blobfs for testing.



~~~{.sh}

test/lib/blobfs/mkfs/mkfs /usr/local/etc/spdk/rocksdb.conf Nvme0n1

~~~



At this point, RocksDB is ready for testing with SPDK.  Three db_bench parameters are used to configure SPDK:

At this point, RocksDB is ready for testing with SPDK.  Three `db_bench` parameters are used to configure SPDK:



1. spdk - Defines the name of the SPDK configuration file.  If omitted, RocksDB will use the default PosixEnv implementation

1. `spdk` - Defines the name of the SPDK configuration file.  If omitted, RocksDB will use the default PosixEnv implementation

   instead of SpdkEnv. (Required)

2. spdk_bdev - Defines the name of the SPDK block device which contains the BlobFS to be used for testing. (Required)

3. spdk_cache_size - Defines the amount of userspace cache memory used by SPDK.  Specified in terms of megabytes (MB).

2. `spdk_bdev` - Defines the name of the SPDK block device which contains the BlobFS to be used for testing. (Required)

3. `spdk_cache_size` - Defines the amount of userspace cache memory used by SPDK.  Specified in terms of megabytes (MB).

   Default is 4096 (4GB).  (Optional)



SPDK has a set of scripts which will run db_bench against a variety of workloads and capture performance and profiling

SPDK has a set of scripts which will run `db_bench` against a variety of workloads and capture performance and profiling

data.  The primary script is `test/blobfs/rocksdb/run_tests.sh`.



# FUSE
@@ -57,7 +76,9 @@ data. The primary script is `test/blobfs/rocksdb/run_tests.sh`. 
BlobFS provides a FUSE plug-in to mount an SPDK BlobFS as a kernel filesystem for inspection or debug purposes.

The FUSE plug-in requires fuse3 and will be built automatically when fuse3 is detected on the system.



~~~{.sh}

test/lib/blobfs/fuse/fuse /usr/local/etc/spdk/rocksdb.conf Nvme0n1 /mnt/fuse

~~~



Note that the FUSE plug-in has some limitations - see the list below.