Commit e189949f authored by Yue-Zhu's avatar Yue-Zhu Committed by Tomasz Zawadzki
Browse files

app: enable configurable msg_mempool_size for reactor 



This patch adds "--msg-mempool-size" option for spdk app to allow
reactors' msg_mempool_size being configurable via commond line.
We tested the rbd_bdev performance for Ceph CTX sharing with high RBD volume count via bdevperf.
When testing with 256 volumes and limited Ceph CTX (e.g., 2 Ceph ctx for 256 volumes,
which are created though bdev_rbd_register_cluster),
error message "the *ERROR*: msg could not be allocated error message"
keeps showing and the bdev_perf program hangs.
We found the issue from the limited msg_mempool_size size, which is hardcoded
by SPDK_DEFAULT_MSG_MEMPOOL_SIZE in thread.h.
Therefore, we enable the "--msg-mempool-size" option to allow configurable msg_mempool_size.

Signed-off-by: default avatarYue-Zhu <yue.zhu@ibm.com>
Change-Id: I54db7fd46247b2f18112bb994ecce6f4b7e5bf9c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15552


Reviewed-by: default avatarShuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: default avatarAleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: default avatarJim Harris <james.r.harris@intel.com>
Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
parent 3a380243

lib/event/app.c

+14 −0
Original line number Diff line number Diff line
@@ -128,6 +128,8 @@ static const struct option g_cmdline_options[] = { 
	{"rpcs-allowed",		required_argument,	NULL, RPCS_ALLOWED_OPT_IDX},

#define ENV_VF_TOKEN_OPT_IDX 269

	{"vfio-vf-token",		required_argument,	NULL, ENV_VF_TOKEN_OPT_IDX},

#define MSG_MEMPOOL_SIZE_OPT_IDX 270

	{"msg-mempool-size",		required_argument,	NULL, MSG_MEMPOOL_SIZE_OPT_IDX},

};



static void
@@ -860,6 +862,8 @@ usage(void (*app_usage)(void)) 
	}

	printf("     --disable-cpumask-locks    Disable CPU core lock files.\n");

	printf("     --silence-noticelog   disable notice level logging to stderr\n");

	printf("     --msg-mempool-size <size>  global message memory pool size in count (default: %d)\n",

	       SPDK_DEFAULT_MSG_MEMPOOL_SIZE);

	printf(" -u, --no-pci              disable PCI access\n");

	printf("     --wait-for-rpc        wait for RPCs to initialize subsystems\n");

	printf("     --max-delay <num>     maximum reactor delay (in microseconds)\n");
@@ -1036,6 +1040,16 @@ spdk_app_parse_args(int argc, char **argv, struct spdk_app_opts *opts, 
			opts->mem_size = (int) mem_size_mb;

			break;

		}

		case MSG_MEMPOOL_SIZE_OPT_IDX:

			tmp = spdk_strtol(optarg, 10);

			if (tmp <= 0) {

				SPDK_ERRLOG("Invalid message memory pool size %s\n", optarg);

				goto out;

			}



			opts->msg_mempool_size = (size_t)tmp;

			break;



		case NO_PCI_OPT_IDX:

			opts->no_pci = true;

			break;