Commit 7b7799bf authored by Jim Harris's avatar Jim Harris
Browse files

fio_plugin: switch to LD_PRELOAD instead of dynamically loading 



fio has a race between reap_threads() and free_ioengine().  free_ioengine()
will call the ioengine's cleanup routine and then dlclose it if it
is dynamically linked (like the spdk fio plugin).  free_ioengine() does
not set td->io_ops = NULL though until after dlclose() is complete.  If
reap_threads() tries to dereference td->io_ops after our plugin has been
closed but before io_ops was set to NULL, it will segfault.

Solution (until an upstream fio fix is available) is to use LD_PRELOAD
instead.

Signed-off-by: default avatarJim Harris <james.r.harris@intel.com>
Change-Id: Ide4eb3cb92a636513289107fc211fdf1f98b616f

Reviewed-on: https://review.gerrithub.io/365272


Tested-by: default avatarSPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: default avatarDaniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: default avatarBen Walker <benjamin.walker@intel.com>
Reviewed-by: default avatarZiye Yang <optimistyzy@gmail.com>
parent c7e5d9f1

examples/nvme/fio_plugin/README.md

+7 −3
Original line number Diff line number Diff line
@@ -27,10 +27,11 @@ as this README. 
Usage

------



To use the SPDK fio plugin with fio, simply set the following in the fio configuration file

(see example_config.fio in the same directory as this README).

To use the SPDK fio plugin with fio, specify the plugin binary using LD_PRELOAD when running

fio and set ioengine=spdk in the fio configuration file (see example_config.fio in the same

directory as this README).



    ioengine=<path to fio_plugin binary>

    LD_PRELOAD=<path to fio_plugin binary> fio



To select NVMe devices, you simply pass an identifier as the filename in the format
@@ -45,6 +46,9 @@ NVMe namespaces start at 1, not 0! And it should be put on the end. For example, 
Currently the SPDK fio plugin is limited to thread usage model, so fio jobs must also specify thread=1

when using the SPDK fio plugin.



fio also currently has a race condition on shutdown if dynamically loading the ioengine by specifying the

engine's full path via the ioengine parameter - LD_PRELOAD is recommended to avoid this race condition.



When testing random workloads, it is recommended to set norandommap=1.  fio's random map

processing consumes extra CPU cycles which will degrade performance over time with

the fio_plugin since all I/O are submitted and completed on a single CPU core.

examples/nvme/fio_plugin/example_config.fio

+1 −1
Original line number Diff line number Diff line 
[global]

ioengine=./examples/nvme/fio_plugin/fio_plugin

ioengine=spdk

thread=1

group_reporting=1

direct=1

examples/nvme/fio_plugin/fio_plugin.c

+11 −1
Original line number Diff line number Diff line
@@ -486,7 +486,7 @@ static void spdk_fio_cleanup(struct thread_data *td) 


/* FIO imports this structure using dlsym */

struct ioengine_ops ioengine = {

	.name			= "spdk_fio",

	.name			= "spdk",

	.version		= FIO_IOOPS_VERSION,

	.queue			= spdk_fio_queue,

	.getevents		= spdk_fio_getevents,
@@ -502,3 +502,13 @@ struct ioengine_ops ioengine = { 
	.io_u_free		= spdk_fio_io_u_free,

	.flags			= FIO_RAWIO | FIO_NOEXTEND | FIO_NODISKUTIL | FIO_MEMALIGN,

};



static void fio_init fio_spdk_register(void)

{

	register_ioengine(&ioengine);

}



static void fio_exit fio_spdk_unregister(void)

{

	unregister_ioengine(&ioengine);

}

examples/nvme/fio_plugin/full_bench.fio

+1 −1
Original line number Diff line number Diff line 
[global]

ioengine=./examples/nvme/fio_plugin/fio_plugin

ioengine=spdk

thread=1

group_reporting=1

direct=1

test/lib/nvme/nvme.sh

+3 −1
Original line number Diff line number Diff line
@@ -45,10 +45,12 @@ timing_enter overhead 
$rootdir/test/lib/nvme/overhead/overhead -s 4096 -t 1

timing_exit overhead



PLUGIN_DIR=$rootdir/examples/nvme/fio_plugin



if [ -d /usr/src/fio ]; then

	timing_enter fio_plugin

	for bdf in $(linux_iter_pci 0108); do

		/usr/src/fio/fio $rootdir/examples/nvme/fio_plugin/example_config.fio --filename="trtype=PCIe traddr=${bdf//:/.} ns=1"

		LD_PRELOAD=$PLUGIN_DIR/fio_plugin /usr/src/fio/fio $PLUGIN_DIR/example_config.fio --filename="trtype=PCIe traddr=${bdf//:/.} ns=1"

		break

	done