Commit 0070858e authored by Michal Berger's avatar Michal Berger Committed by Tomasz Zawadzki
Browse files

scripts/setup.sh: Use HUGE_EVEN_ALLOC logic by default 



To that end, remove it altogether and allow setup.sh to always split
requested amount of hugepages across all available nodes. Custom
setups per node are still available through the HUGENODE var.

Adjust some of the hugepages.sh tests to adhere to the new default.
This change allows us to remove the per_node_1G_alloc() test since
its flow is now covered by the custom_alloc().

By default, autotest.sh sets HUGEMEM=4096 and by spreading it across
all the nodes, for the minimal scenario, we get only 2GB on a single
node. For vhost tests, the default alloc per VM via vm_setup() is set
to 1GB so sharing that 2GB between qemu and SPDK spreads it a bit too
thin.

Case in point, for vhost.vhost_blk_packed_ring_integrity test roughly
> 600 hugepages is used for vhost. Since it slurps everything from
node0, only < 512 remains. Since VMs are assigned per node, the most
basic setups keep VM_*_qemu_numa_node set to 0. This is then used by
qemu to bind to a specific node. With 1G, < 512 hugepages is simply
not enough.

With that in mind, for vhost tests, keep all allocations on a single
node instead of trying to figure out right amount of memory per node
to keep the old behavior in (this may change in the future when
SPDK/vhost becomes numa-aware).

Change-Id: I83e18bfa4cc6de0a777804b354de083ae6ae9d8c
Signed-off-by: default avatarMichal Berger <michal.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/24176


Community-CI: Mellanox Build Bot
Reviewed-by: default avatarTomasz Zawadzki <tomasz@tzawadzki.com>
Reviewed-by: default avatarJim Harris <jim.harris@samsung.com>
Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
parent 34edd9f1

scripts/common/setup/interactive.sh

+0 −3
Original line number Diff line number Diff line
@@ -301,7 +301,6 @@ hugepages() { 
	[[ $os == Linux ]] || return 0

	local hp



	HUGE_EVEN_ALLOC=no

	while read -rp "('clear' 'even' 'commit' HUGEMEM[=$HUGEMEM MB])> " hp; do

		hp=${hp,,}

		if [[ -z $hp ]]; then
@@ -309,8 +308,6 @@ hugepages() { 
		elif [[ $hp == clear ]]; then

			clear_hugepages

			return

		elif [[ $hp == even ]]; then

			HUGE_EVEN_ALLOC=yes

		elif [[ $hp =~ ^[1-9][0-9]*$ ]]; then

			NRHUGE=""

			HUGEMEM=$hp

scripts/setup.sh

+2 −6
Original line number Diff line number Diff line
@@ -51,9 +51,6 @@ function usage() { 
	echo "HUGEMEM           Size of hugepage memory to allocate (in MB). 2048 by default."

	echo "                  For NUMA systems, the hugepages will be distributed on node0 by"

	echo "                  default."

	echo "HUGE_EVEN_ALLOC   If set to 'yes', hugepages will be evenly distributed across all"

	echo "                  system's NUMA nodes (effectively ignoring anything set in HUGENODE)."

	echo "                  Uses kernel's default for hugepages size."

	echo "NRHUGE            Number of hugepages to allocate. This variable overwrites HUGEMEM."

	echo "HUGENODE          Specific NUMA node to allocate hugepages on. Multiple nodes can be"

	echo "                  separated with comas. By default, NRHUGE will be applied on each node."
@@ -64,7 +61,7 @@ function usage() { 
	echo "                  setting is used."

	echo "SHRINK_HUGE       If set to 'yes', hugepages allocation won't be skipped in case"

	echo "                  number of requested hugepages is lower from what's already"

	echo "                  allocated. Doesn't apply when HUGE_EVEN_ALLOC is in use."

	echo "                  allocated."

	echo "CLEAR_HUGE        If set to 'yes', the attempt to remove hugepages from all nodes will"

	echo "                  be made prior to allocation".

	echo "PCI_ALLOWED"
@@ -553,8 +550,7 @@ configure_linux_hugepages() { 
		clear_hugepages

	fi



	if [[ $HUGE_EVEN_ALLOC == yes ]]; then

		clear_hugepages

	if [[ -z $HUGENODE ]]; then

		check_hugepages_alloc /proc/sys/vm/nr_hugepages

		return 0

	fi

test/common/autotest_common.sh

+0 −6
Original line number Diff line number Diff line
@@ -271,12 +271,6 @@ fi 
if [ "$(uname -s)" = "Linux" ]; then

	HUGEMEM=${HUGEMEM:-4096}

	export CLEAR_HUGE=yes

	if [[ $SPDK_TEST_CRYPTO -eq 1 || $SPDK_TEST_VBDEV_COMPRESS -eq 1 ]]; then

		# Make sure that memory is distributed across all NUMA nodes - by default, all goes to

		# node0, but if QAT devices are attached to a different node, all of their VFs will end

		# up under that node too and memory needs to be available there for the tests.

		export HUGE_EVEN_ALLOC=yes

	fi



	MAKE="make"

	MAKEFLAGS=${MAKEFLAGS:--j$(nproc)}

test/openstack/run_openstack_tests.sh

+1 −1
Original line number Diff line number Diff line
@@ -13,7 +13,7 @@ source $rootdir/test/nvmf/common.sh 


rpc_py=$rootdir/scripts/rpc.py



HUGE_EVEN_ALLOC=yes HUGEMEM=1024 nvmftestinit

HUGEMEM=1024 nvmftestinit



function finish_test() {

	{

test/scheduler/common.sh

+6 −0
Original line number Diff line number Diff line
@@ -152,6 +152,12 @@ is_cpu_offline_f() { 
	! is_cpu_online_f "$1"

}



is_numa() {

	local nodes=("$sysfs_node/node"+([0-9]))



	((${#nodes[@]} > 1))

}



online_cpu() {

	is_cpu_offline_f "$1" || return 0

	echo 1 > "$sysfs_cpu/cpu$1/online"