Commit 904ac49f authored by Michal Berger's avatar Michal Berger Committed by Jim Harris
Browse files

scripts/setup: Configure binding of the controllers in parallel 

There are some devices for which nvme driver takes a long time to
finalize the unbind stage. With that in mind, each device would
add up a significant amount of time needed for setup.sh to complete.

To mitigate such a scenario, make sure the controllers are unbound
in a parallel fashion.

Examples taken from the system with 19 nvmes on board:

[root@supermicro4 spdk]# time ./scripts/setup.sh &>/dev/null

real    0m36.250s
user    0m1.024s
sys     0m1.990s

[root@supermicro4 spdk]# time ./scripts/setup.sh &>/dev/null

real    0m4.848s
user    0m0.867s
sys     0m17.605s

Also, take note that this is currently done only for the nvme
devices since other, i.e., ioatdma, seem to trigger a BUG in the
kernel when unbound in parallel. Some details here:

  https://bugzilla.kernel.org/show_bug.cgi?id=209041



Change-Id: Icaeb2b2ecb306f149587bc5da73743b1519bc5d6
Signed-off-by: default avatarMichal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3893


Community-CI: Mellanox Build Bot
Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: default avatarJim Harris <james.r.harris@intel.com>
Reviewed-by: default avatarTomasz Zawadzki <tomasz.zawadzki@intel.com>
parent 5ea54946

scripts/setup.sh

+11 −1
Original line number Diff line number Diff line
@@ -274,9 +274,19 @@ function configure_linux_pci() { 


	for bdf in "${!all_devices_d[@]}"; do

		if ((all_devices_d["$bdf"] == 0)); then

			if [[ -n ${nvme_d["$bdf"]} ]]; then

				# Some nvme controllers may take significant amount of time while being

				# unbound from the driver. Put that task into background to speed up the

				# whole process. Currently this is done only for the devices bound to the

				# nvme driver as other, i.e., ioatdma's, trigger a kernel BUG when being

				# unbound in parallel. See https://bugzilla.kernel.org/show_bug.cgi?id=209041.

				linux_bind_driver "$bdf" "$driver_name" &

			else

				linux_bind_driver "$bdf" "$driver_name"

			fi

		fi

	done

	wait



	echo "1" > "/sys/bus/pci/rescan"

}