Commit 1bed9c1f authored by Tomasz Zawadzki's avatar Tomasz Zawadzki Committed by Darek Stojaczyk
Browse files

test/asan: preload ASAN for fio 



ASAN needs to be LD_PRELOADed before SPDK fio_plugin
in order to analyze its code.
Just adding that will report any issues in fio binary as
well as the fio_plugin.
To prevent known fio leaks from affecting the results,
a suppression list for LeakSanitizer (used in conjunction with ASAN).

At this time the suppression list contains known leaks
for fio 3.3. The list might need adjustments as fio
version is updated.

Side note. Even though it is possible to specify directory
to ignore ("leak:/usr/src/fio/"). Which in theory should
suppress any leaks in fio. It has side effect of hiding
SPDK leaks as well, since the fio_plugins leaks are
seen as coming from /usr/src/fio/ioengines.c.

See below for examples of each suppressed error:

Direct leak of 42 byte(s) in 4 object(s) allocated from:
    #0 0x7f9d52f3e320 in strdup (/lib64/libasan.so.5+0x3b320)
    #1 0x41f267 in get_new_job /usr/src/fio/init.c:490

Direct leak of 914936 byte(s) in 10397 object(s) allocated from:
    #0 0x7f74422e8ea6 in __interceptor_calloc (/lib64/libasan.so.5+0x10dea6)
    #1 0x46402e in log_io_piece /usr/src/fio/iolog.c:214

Direct leak of 608 byte(s) in 19 object(s) allocated from:
    #0 0x7f74422e8ca8 in __interceptor_malloc (/lib64/libasan.so.5+0x10dca8)
    #1 0x44c4e1 in add_to_dump_list /usr/src/fio/parse.c:1039
    #2 0x44c4e1 in parse_option /usr/src/fio/parse.c:1098

Direct leak of 173 byte(s) in 20 object(s) allocated from:
    #0 0x7f744227153d in strdup (/lib64/libasan.so.5+0x9653d)
    #1 0x44b50d in __handle_option /usr/src/fio/parse.c:718

Indirect leak of 111925528 byte(s) in 1271881 object(s) allocated from:
    #0 0x7f74422e8ea6 in __interceptor_calloc (/lib64/libasan.so.5+0x10dea6)
    #1 0x46402e in log_io_piece /usr/src/fio/iolog.c:214

Indirect leak of 171 byte(s) in 19 object(s) allocated from:
    #0 0x7f744227153d in strdup (/lib64/libasan.so.5+0x9653d)
    #1 0x44c4ed in add_to_dump_list /usr/src/fio/parse.c:1040
    #2 0x44c4ed in parse_option /usr/src/fio/parse.c:1098

Indirect leak of 167 byte(s) in 19 object(s) allocated from:
    #0 0x7f744227153d in strdup (/lib64/libasan.so.5+0x9653d)
    #1 0x44c502 in add_to_dump_list /usr/src/fio/parse.c:1042
    #2 0x44c502 in parse_option /usr/src/fio/parse.c:1098

Change-Id: I9b5811993508421be50b12af160645c77ea93d7e
Signed-off-by: default avatarTomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456315


Reviewed-by: default avatarBen Walker <benjamin.walker@intel.com>
Reviewed-by: default avatarDarek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
parent 18b504af

test/bdev/blockdev.sh

+1 −1
Original line number Diff line number Diff line
@@ -112,7 +112,7 @@ bdevs_name=$(echo $bdevs | jq -r '.name') 
nbd_function_test $testdir/bdev.conf "$bdevs_name"

timing_exit nbd



if [ -d /usr/src/fio ] && [ $SPDK_RUN_ASAN -eq 0 ]; then

if [ -d /usr/src/fio ]; then

	timing_enter fio



	timing_enter fio_rw_verify

test/common/autotest_common.sh

+13 −2
Original line number Diff line number Diff line
@@ -81,6 +81,11 @@ sudo rm -rf "$asan_suppression_file" 
# LSAN_OPTIONS.

echo "leak:spdk_fs_alloc_thread_ctx" >> "$asan_suppression_file"



# Suppress known leaks in fio project

echo "leak:/usr/src/fio/parse.c" >> "$asan_suppression_file"

echo "leak:/usr/src/fio/iolog.c" >> "$asan_suppression_file"

echo "leak:/usr/src/fio/init.c" >> "$asan_suppression_file"



export LSAN_OPTIONS=suppressions="$asan_suppression_file"



export DEFAULT_RPC_ADDR="/var/tmp/spdk.sock"
@@ -771,7 +776,10 @@ function fio_bdev() 
	local fio_dir="/usr/src/fio"

	local bdev_plugin="$rootdir/examples/bdev/fio_plugin/fio_plugin"



	LD_PRELOAD="$bdev_plugin" "$fio_dir"/fio "$@"

	# Preload AddressSanitizer library to fio if fio_plugin was compiled with it

	local asan_lib=$(ldd $bdev_plugin | grep libasan | awk '{print $3}')



	LD_PRELOAD=""$asan_lib" "$bdev_plugin"" "$fio_dir"/fio "$@"

}



function fio_nvme()
@@ -780,7 +788,10 @@ function fio_nvme() 
	local fio_dir="/usr/src/fio"

	local nvme_plugin="$rootdir/examples/nvme/fio_plugin/fio_plugin"



	LD_PRELOAD="$nvme_plugin" "$fio_dir"/fio "$@"

	# Preload AddressSanitizer library to fio if fio_plugin was compiled with it

	asan_lib=$(ldd $nvme_plugin | grep libasan | awk '{print $3}')



	LD_PRELOAD=""$asan_lib" "$nvme_plugin"" "$fio_dir"/fio "$@"

}



function get_lvs_free_mb()

test/nvme/nvme.sh

+1 −2
Original line number Diff line number Diff line
@@ -162,8 +162,7 @@ if [ $(uname) = Linux ]; then 
	kill_stub

fi



if [ -d /usr/src/fio ] && [ $SPDK_RUN_ASAN -eq 0 ]; then

	# Only test when ASAN is not enabled. If ASAN is enabled, we cannot test.

if [ -d /usr/src/fio ]; then

	timing_enter fio_plugin

	PLUGIN_DIR=$rootdir/examples/nvme/fio_plugin

	for bdf in $(iter_pci_class_code 01 08 02); do

test/nvmf/nvmf.sh

+1 −3
Original line number Diff line number Diff line
@@ -58,9 +58,7 @@ run_test suite test/nvmf/host/perf.sh $TEST_ARGS --transport=tcp 
# TODO: disabled due to intermittent failures (RDMA_CM_EVENT_UNREACHABLE/ETIMEDOUT)

#run_test test/nvmf/host/identify_kernel_nvmf.sh $TEST_ARGS

run_test suite test/nvmf/host/aer.sh $TEST_ARGS

if [ $SPDK_RUN_ASAN -eq 0 ]; then

run_test suite test/nvmf/host/fio.sh $TEST_ARGS

fi



timing_exit host