Commit c8a36b04 authored by Jim Harris's avatar Jim Harris Committed by Tomasz Zawadzki
Browse files

test/iscsi_tgt: run most Calsoft tests serially 



We currently use a hack AllowDuplicatedIsid with the
iSCSI target to allow running multiple Calsoft tests
in parallel, since each Calsoft test always uses the
same InitiatorName.

But this causes some tests to intermittently fail,
likely due to changed behavior with many sessions
sharing the same InitiatorName.  The problem is
very difficult to reproduce, and made worse since
the Calsoft tests themselves are binary only.

So instead split the tests into two sets - one set
contains nopin-related tests which each take longer
to run due to various interval expirations needed for
the test.  There are just a few of these tests, and
do not appear to ever cause an intermittent failure,
so we run these in parallel.  The rest are all run
serially and run rather quickly.

Fixes issue #2416.

Signed-off-by: default avatarJim Harris <james.r.harris@intel.com>
Change-Id: I0bd29da8a71d4c7593bba11f1e2f2e11e2289d8a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16022


Reviewed-by: default avatarMichal Berger <michal.berger@intel.com>
Reviewed-by: default avatarTomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: default avatarShuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: default avatarKamil Godzwon <kamilx.godzwon@intel.com>
Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
parent c6951560

test/iscsi_tgt/calsoft/calsoft.py

+25 −15
Original line number Diff line number Diff line
@@ -72,6 +72,9 @@ def main(): 


    all_cases = [x for x in os.listdir(CALSOFT_BIN_PATH) if x.startswith('tc')]

    all_cases.sort()

    nopin_cases = ['tc_err_2_4', 'tc_err_8_2', 'tc_ffp_7_6_1', 'tc_ffp_7_7_3', 'tc_ffp_7_7_2',

                   'tc_ffp_7_7_5', 'tc_ffp_7_6_4', 'tc_ffp_7_6_3', 'tc_ffp_7_7_4', 'tc_ffp_7_6_2',

                   'tc_ffp_7_7_1', 'tc_ffp_7_7']



    case_result_list = []
@@ -84,22 +87,29 @@ def main(): 
        case_result_list.append({"Name": case, "Result": "SKIP"})



    thread_objs = []

    left_cases = list(set(all_cases) - set(known_failed_cases))

    index = 0

    max_thread_count = 32



    while index < len(left_cases):

        cur_thread_count = 0

        for thread_obj in thread_objs:

    # The Calsoft tests all pull their InitiatorName from the its.conf file.  We set the

    # AllowDuplicatedIsid flag in the SPDK JSON config, to allow these tests to run in

    # parallel where needed, but we only run tests in parallel that make sense - in this case

    # only the nopin-related tests which take longer to run because of various timeouts.

    serial_cases = list(set(all_cases) - set(known_failed_cases) - set(nopin_cases))

    parallel_cases = nopin_cases



    for test_case in serial_cases:

        thread_obj = threading.Thread(target=run_case, args=(test_case, case_result_list, log_dir, ))

        thread_obj.start()

        thread_obj.join(30)

        if thread_obj.is_alive():

                cur_thread_count += 1

        while cur_thread_count < max_thread_count and index < len(left_cases):

            thread_obj = threading.Thread(target=run_case, args=(left_cases[index], case_result_list, log_dir, ))

            # Thread is still alive, meaning the join() timeout expired.

            print("Thread timeout")

            exit(1)



    for test_case in parallel_cases:

        thread_obj = threading.Thread(target=run_case, args=(test_case, case_result_list, log_dir, ))

        thread_obj.start()

        time.sleep(0.02)

        thread_objs.append(thread_obj)

            index += 1

            cur_thread_count += 1



    end_time = time.time() + 30

    while time.time() < end_time:

        for thread_obj in thread_objs: