Commit a0e22e13 authored by paul luse's avatar paul luse Committed by Jim Harris
Browse files

test: add 'match' util and use it with blobstore 



Will follow this up with a doc change but want to make sure we're
all good with it first. This is meant to not only beef up
blobstore testing but provide the 'match' utilitiy for all test
cases where we are currently calling an executable and either
counting only on a return code to determine success or failure
or worse yet we're just running it and if it doesn't explode we
assume its a pass.

The 'match' util was borrowed from the PMDK folks after first
adding the "ignore" feature upstream to make it easier to use
in SPDK.  It works like this:

When the developer checks in a test they create and check in
the output of the test with two different file extensions:

.ignore: should include a string per line for output lines
that we want to totally ignore typically because they're
platform specific so the output could be different from
machine to machine.  In this case I'm ignoring all output
lines with 'DPDK' or 'EAL' or '...' in them. The first
few are obvious, the last is because the test tool will
print a varrying number of these as progress indicators.

.match: this is a copy of the output that the developer
'fixes' up by replacing platform specific output strings
with replacable tokens as described in the 'match' help.
This is where you'd want to match an entire line minus
something like a CPU count or free block count or
something. The 'ignore' feature was added simply so we
wouldn't have to edit every single line of an output
file that had DPDK or EAL in it.

Then you modify the test script to save the output and
smply run the match util providing the name of the
match file and if it fails to match the actual output
with the saved output that's been token'ized the script
will error.

The obvious advantage here is that now we can confirm all
of the output from a test executable is as we expect.

Change-Id: I701d36ee83d37b6193e16ed3171e114f814e5eb3
Signed-off-by: default avatarpaul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/397027


Tested-by: default avatarSPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: default avatarDaniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: default avatarJim Harris <james.r.harris@intel.com>
parent 20cb585b

examples/blob/cli/blobcli.c

+1 −1
Original line number Diff line number Diff line
@@ -701,7 +701,7 @@ dump_imp_open_cb(void *cb_arg, struct spdk_blob *blob, int bserrno) 
	} else {

		cli_context->fp = fopen(cli_context->file, "r");

		if (cli_context->fp == NULL) {

			printf("Error in opening file\n");

			printf("Error in opening file: errno %d\n", errno);

			spdk_blob_close(cli_context->blob, close_cb, cli_context);

			return;

		}

test/app/match/match

0 → 100755
+326 −0
Original line number Diff line number Diff line 
#!/usr/bin/env perl

#

# Copyright 2014-2017, Intel Corporation

#

# Redistribution and use in source and binary forms, with or without

# modification, are permitted provided that the following conditions

# are met:

#

#     * Redistributions of source code must retain the above copyright

#       notice, this list of conditions and the following disclaimer.

#

#     * Redistributions in binary form must reproduce the above copyright

#       notice, this list of conditions and the following disclaimer in

#       the documentation and/or other materials provided with the

#       distribution.

#

#     * Neither the name of the copyright holder nor the names of its

#       contributors may be used to endorse or promote products derived

#       from this software without specific prior written permission.

#

# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS

# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT

# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR

# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT

# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,

# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT

# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,

# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY

# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT

# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE

# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

#



#

# match -- compare an output file with expected results

#

# usage: match [-adoqv] [match-file]...

#

# this script compares the output from a test run, stored in a file, with

# the expected output.  comparison is done line-by-line until either all

# lines compare correctly (exit code 0) or a miscompare is found (exit

# code nonzero).

#

# expected output is stored in a ".match" file, which contains a copy of

# the expected output with embedded tokens for things that should not be

# exact matches.  the supported tokens are:

#

#	$(N)	an integer (i.e. one or more decimal digits)

#	$(NC)	one or more decimal digits with comma separators

#	$(FP)	a floating point number

#	$(S)	ascii string

#	$(X)	hex number

#	$(XX)	hex number prefixed with 0x

#	$(W)	whitespace

#	$(nW)	non-whitespace

#	$(*)	any string

#	$(DD)	output of a "dd" run

#	$(OPT)	line is optional (may be missing, matched if found)

#	$(OPX)	ends a contiguous list of $(OPT)...$(OPX) lines, at least

#		one of which must match

#

# Additionally, if any "X.ignore" file exists, strings or phrases found per

# line in the file will be ignored if found as a substring in the

# corresponding output file (making it easy to skip entire output lines).

#

# arguments are:

#

#	-a	find all files of the form "X.match" in the current

#		directory and match them again the corresponding file "X".

#

#	-o	custom output filename - only one match file can be given

#

#	-d	debug -- show lots of debug output

#

#	-q	don't print log files on mismatch

#

#	-v	verbose -- show every line as it is being matched

#



use strict;

use Getopt::Std;

use Encode;

use v5.16;



select STDERR;

binmode(STDOUT, ":utf8");

binmode(STDERR, ":utf8");



my $Me = $0;

$Me =~ s,.*/,,;



our ($opt_a, $opt_d, $opt_q, $opt_v, $opt_o);



$SIG{HUP} = $SIG{INT} = $SIG{TERM} = $SIG{__DIE__} = sub {

	die @_ if $^S;

	my $errstr = shift;

	die "FAIL: $Me: $errstr";

};



sub usage {

	my $msg = shift;



	warn "$Me: $msg\n" if $msg;

	warn "Usage: $Me [-adqv] [match-file]...\n";

	warn "   or: $Me [-dqv] -o output-file match-file...\n";

	exit 1;

}



getopts('adoqv') or usage;



my %match2file;



if ($opt_a) {

	usage("-a and filename arguments are mutually exclusive")

		if $#ARGV != -1;

	opendir(DIR, '.') or die "opendir: .: $!\n";

	my @matchfiles = grep { /(.*)\.match$/ && -f $1 } readdir(DIR);

	closedir(DIR);

	die "no files found to process\n" unless @matchfiles;

	foreach my $mfile (@matchfiles)  {

		die "$mfile: $!\n" unless open(F, $mfile);

		close(F);

		my $ofile = $mfile;

		$ofile =~ s/\.match$//;

		die "$mfile found but cannot open $ofile: $!\n"

			unless open(F, $ofile);

		close(F);

		$match2file{$mfile} = $ofile;

	}

} elsif ($opt_o) {

	usage("-o argument requires two paths") if $#ARGV != 1;



	$match2file{$ARGV[1]} = $ARGV[0];

} else {

	usage("no match-file arguments found") if $#ARGV == -1;



	# to improve the failure case, check all filename args exist and

	# are provided in pairs now, before going through and processing them

	foreach my $mfile (@ARGV) {

		my $ofile = $mfile;

		usage("$mfile: not a .match file") unless

			$ofile =~ s/\.match$//;

		usage("$mfile: $!") unless open(F, $mfile);

		close(F);

		usage("$ofile: $!") unless open(F, $ofile);

		close(F);

		$match2file{$mfile} = $ofile;

	}

}



my $mfile;

my $ofile;

my $ifile;

print "Files to be processed:\n" if $opt_v;

foreach $mfile (sort keys %match2file) {

	$ofile = $match2file{$mfile};

	$ifile = $ofile . ".ignore";

	$ifile = undef unless (-f $ifile);

	if ($opt_v) {

		print "        match-file \"$mfile\" output-file \"$ofile\"";

		if ($ifile) {

			print " ignore-file $ifile\n";

		} else {

			print "\n";

		}

	}

	match($mfile, $ofile, $ifile);

}



exit 0;



#

# strip_it - user can optionally ignore lines from files that contain

# any number of substrings listed in a file called "X.ignore" where X

# is the name of the output file.

#

sub strip_it {

	my ($ifile, $file, $input) = @_;

	# if there is no ignore file just return unaltered input

	return $input unless $ifile;

	my @lines_in = split /^/, $input;

	my $output;

	my $line_in;

	my @i_file = split /^/, snarf($ifile);

	my $i_line;

	my $ignore_it = 0;



	foreach $line_in (@lines_in) {

		my @i_lines = @i_file;

		foreach $i_line (@i_lines) {

			chop($i_line);

			if (index($line_in, $i_line) != -1) {

				$ignore_it = 1;

				if ($opt_v) {

					print "Ignoring (from $file): $line_in";

				}

			}

		}

		if ($ignore_it == 0) {

			$output .= $line_in;

		}

		$ignore_it = 0;

	}

	return $output;

}



#

# match -- process a match-file, output-file pair

#

sub match {

	my ($mfile, $ofile, $ifile) = @_;

	my $pat;

	my $output = snarf($ofile);

	$output = strip_it($ifile, $ofile, $output);

	my $all_lines = $output;

	my $line_pat = 0;

	my $line_out = 0;

	my $opt = 0;

	my $opx = 0;

	my $opt_found = 0;

	my $fstr = snarf($mfile);

	$fstr = strip_it($ifile, $mfile, $fstr);

	for (split /^/, $fstr) {

		$pat = $_;

		$line_pat++;

		$line_out++;

		s/([*+?|{}.\\^\$\[()])/\\$1/g;

		s/\\\$\\\(FP\\\)/[-+]?\\d*\\.?\\d+([eE][-+]?\\d+)?/g;

		s/\\\$\\\(N\\\)/[-+]?\\d+/g;

		s/\\\$\\\(NC\\\)/[-+]?\\d+(,[0-9]+)*/g;

		s/\\\$\\\(\\\*\\\)/\\p{Print}*/g;

		s/\\\$\\\(S\\\)/\\P{IsC}+/g;

		s/\\\$\\\(X\\\)/\\p{XPosixXDigit}+/g;

		s/\\\$\\\(XX\\\)/0x\\p{XPosixXDigit}+/g;

		s/\\\$\\\(W\\\)/\\p{Blank}*/g;

		s/\\\$\\\(nW\\\)/\\p{Graph}*/g;

		s/\\\$\\\(DD\\\)/\\d+\\+\\d+ records in\n\\d+\\+\\d+ records out\n\\d+ bytes \\\(\\d+ .B\\\) copied, [.0-9e-]+[^,]*, [.0-9]+ .B.s/g;

		if (s/\\\$\\\(OPT\\\)//) {

			$opt = 1;

		} elsif (s/\\\$\\\(OPX\\\)//) {

			$opx = 1;

		} else {

			$opt_found = 0;

		}



		if ($opt_v) {

			my @lines = split /\n/, $output;

			my $line;

			if (@lines) {

				$line = $lines[0];

			} else {

				$line = "[EOF]";

			}



			printf("%s:%-3d %s%s:%-3d       %s\n", $mfile, $line_pat, $pat, $ofile, $line_out, $line);

		}



		print " => /$_/\n" if $opt_d;

		print " [$output]\n" if $opt_d;

		unless ($output =~ s/^$_//) {

			if ($opt || ($opx && $opt_found)) {

				printf("%s:%-3d      [skipping optional line]\n", $ofile, $line_out) if $opt_v;

				$line_out--;

				$opt = 0;

			} else {

				if (!$opt_v) {

					if ($opt_q) {

						print "[MATCHING FAILED]\n";

					} else {

						print "[MATCHING FAILED, COMPLETE FILE ($ofile) BELOW]\n$all_lines\n[EOF]\n";

					}

					$opt_v = 1;

					match($mfile, $ofile);

				}



				die "$mfile:$line_pat did not match pattern\n";

			}

		} elsif ($opt) {

			$opt_found = 1;

		}

		$opx = 0;

	}



	if ($output ne '') {

		if (!$opt_v) {

			if ($opt_q) {

				print "[MATCHING FAILED]\n";

			} else {

				print "[MATCHING FAILED, COMPLETE FILE ($ofile) BELOW]\n$all_lines\n[EOF]\n";

			}

		}



		# make it a little more print-friendly...

		$output =~ s/\n/\\n/g;

		die "line $line_pat: unexpected output: \"$output\"\n";

	}

}





#

# snarf -- slurp an entire file into memory

#

sub snarf {

	my ($file) = @_;

	my $fh;

	open($fh, '<', $file) or die "$file $!\n";



	local $/;

	$_ = <$fh>;

	close $fh;



	# check known encodings or die

	my $decoded;

	my @encodings = ("UTF-8", "UTF-16", "UTF-16LE", "UTF-16BE");



	foreach my $enc (@encodings) {

		eval { $decoded = decode( $enc, $_, Encode::FB_CROAK ) };



		if (!$@) {

			$decoded =~ s/\R/\n/g;

			return $decoded;

		}

	}



	die "$Me: ERROR: Unknown file encoding";

}

test/blobstore/blobstore.sh

+17 −3
Original line number Diff line number Diff line
@@ -11,7 +11,6 @@ set -xe 
testdir=$(readlink -f $(dirname $0))

rootdir=$(readlink -f $testdir/../..)

source $rootdir/scripts/autotest_common.sh



timing_enter blobstore



set -e
@@ -19,9 +18,24 @@ set -e 
# Nvme0 target configuration

$rootdir/scripts/gen_nvme.sh > $testdir/blobcli.conf



$rootdir/examples/blob/cli/blobcli -c $testdir/blobcli.conf -b Nvme0n1 -T $testdir/test.bs

# generate random data file for import/export diff

dd if=/dev/urandom of=$testdir/test.pattern bs=1M count=1



cd $testdir

$rootdir/examples/blob/cli/blobcli -c $testdir/blobcli.conf -b Nvme0n1 -T $testdir/test.bs > $testdir/btest.out

cd -



# the tool leaves some trailing whitespaces that we need to strip out

sed -i 's/[[:space:]]*$//' $testdir/btest.out



# the test script will import the test pattern generated by dd and then export

# it to a file so we can compare and confirm basic read and write

$rootdir/test/app/match/match -v $testdir/btest.out.match

diff $testdir/test.pattern $testdir/test.pattern.blob



rm -rf $testdir/btest.out

rm -rf $testdir/blobcli.conf

rm -rf ./*.blob

rm -rf $testdir/*.blob

rm -rf $testdir/test.pattern



timing_exit blobstore

test/blobstore/btest.out.ignore

0 → 100644
+3 −0
Original line number Diff line number Diff line 
DPDK

EAL

....

test/blobstore/btest.out.match

0 → 100644
+123 −0
Original line number Diff line number Diff line 
Starting DPDK 17.11.0 initialization...

[ DPDK EAL parameters: blobcli -c 0x1 --file-prefix=spdk_pid5072 ]

EAL: Probing VFIO support...

EAL: PCI device 0000:00:04.0 on NUMA socket 0

EAL:   probe driver: 8086:2f20 spdk_ioat

EAL: PCI device 0000:00:04.1 on NUMA socket 0

EAL:   probe driver: 8086:2f21 spdk_ioat

EAL: PCI device 0000:00:04.2 on NUMA socket 0

EAL:   probe driver: 8086:2f22 spdk_ioat

EAL: PCI device 0000:00:04.3 on NUMA socket 0

EAL:   probe driver: 8086:2f23 spdk_ioat

EAL: PCI device 0000:00:04.4 on NUMA socket 0

EAL:   probe driver: 8086:2f24 spdk_ioat

EAL: PCI device 0000:00:04.5 on NUMA socket 0

EAL:   probe driver: 8086:2f25 spdk_ioat

EAL: PCI device 0000:00:04.6 on NUMA socket 0

EAL:   probe driver: 8086:2f26 spdk_ioat

EAL: PCI device 0000:00:04.7 on NUMA socket 0

EAL:   probe driver: 8086:2f27 spdk_ioat

EAL: PCI device 0000:80:04.0 on NUMA socket 1

EAL:   probe driver: 8086:2f20 spdk_ioat

EAL: PCI device 0000:80:04.1 on NUMA socket 1

EAL:   probe driver: 8086:2f21 spdk_ioat

EAL: PCI device 0000:80:04.2 on NUMA socket 1

EAL:   probe driver: 8086:2f22 spdk_ioat

EAL: PCI device 0000:80:04.3 on NUMA socket 1

EAL:   probe driver: 8086:2f23 spdk_ioat

EAL: PCI device 0000:80:04.4 on NUMA socket 1

EAL:   probe driver: 8086:2f24 spdk_ioat

EAL: PCI device 0000:80:04.5 on NUMA socket 1

EAL:   probe driver: 8086:2f25 spdk_ioat

EAL: PCI device 0000:80:04.6 on NUMA socket 1

EAL:   probe driver: 8086:2f26 spdk_ioat

EAL: PCI device 0000:80:04.7 on NUMA socket 1

EAL:   probe driver: 8086:2f27 spdk_ioat

EAL: PCI device 0000:06:00.0 on NUMA socket 0

EAL:   probe driver: 8086:953 spdk_nvme



SCRIPT NOW PROCESSING: -i

Init blobstore using bdev Product Name: NVMe disk

blobstore init'd: ($(XX))



SCRIPT NOW PROCESSING: -l bdevs



List bdevs:

	bdev Name: Nvme0n1

	bdev Product Name: NVMe disk





SCRIPT NOW PROCESSING: -n 1

New blob id 4294967296

blob now has USED clusters of 1



SCRIPT NOW PROCESSING: -p $B0

Super Blob ID has been set.



SCRIPT NOW PROCESSING: -n 1

New blob id 4294967297

blob now has USED clusters of 1



SCRIPT NOW PROCESSING: -m $B1 test.pattern

Working...............................................................................................................................................................................................................................................................

Blob import complete (from test.pattern).



SCRIPT NOW PROCESSING: -d $B1 test.pattern.blob

Working................................................................................................................................................................................................................................................................

File write complete (to test.pattern.blob).



SCRIPT NOW PROCESSING: -x $B1 key val

Xattr has been set.



SCRIPT NOW PROCESSING: -s bs

Blobstore Public Info:

	Using bdev Product Name: NVMe disk

	API Version: $(N)

	super blob ID: 4294967296

	page size: 4096

	cluster size: 1048576

	# free clusters: $(N)

	blobstore type:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................



Blobstore Private Info:

	Metadata start (pages): $(N)

	Metadata length (pages): $(N)



SCRIPT NOW PROCESSING: -s $B1

Blob Public Info:

blob ID: 4294967297

# of clusters: 1

# of bytes: 1048576

# of pages: 256

# of xattrs: 1

xattrs:



(0) Name:key

(0) Value:



00000000  76 61 6c 00                                       val.



Blob Private Info:

state: CLEAN

open ref count: 1



SCRIPT NOW PROCESSING: -r $B1 key

Xattr has been removed.



SCRIPT NOW PROCESSING: -s bs

Blobstore Public Info:

	Using bdev Product Name: NVMe disk

	API Version: 3

	super blob ID: 4294967296

	page size: 4096

	cluster size: 1048576

	# free clusters: $(N)

	blobstore type:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................



Blobstore Private Info:

	Metadata start (pages): $(N)

	Metadata length (pages): $(N)



SCRIPT NOW PROCESSING: -X