Commit 3092c61d authored by Jim Harris's avatar Jim Harris
Browse files

nvmf: enable dynamic buf_cache_size calculation 



Allow transports to specify a default UINT32_MAX
as the buf_cache_size. If user does not override this
when creating the transport, calculate the buf_cache_size
dynamically using the number of poll groups and the
size of the buffer pool (num_shared_buffers).  We will
allocate 75% of the buffers for the caches, meaning
the buf_cache_size will be calculated as:

(num_shared_buffers * 3 / 4) / num_poll_groups

Signed-off-by: default avatarJim Harris <james.r.harris@intel.com>
Change-Id: I97768aea701060bbe0ff1925e5322229fa8d051c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17334


Community-CI: Mellanox Build Bot
Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: default avatarBen Walker <benjamin.walker@intel.com>
Reviewed-by: default avatarAleksey Marchuk <alexeymar@nvidia.com>
parent 280a3abc

lib/nvmf/transport.c

+18 −0
Original line number Diff line number Diff line
@@ -557,6 +557,24 @@ nvmf_transport_poll_group_create(struct spdk_nvmf_transport *transport, 
	}



	tgroup->buf_cache_size = transport->opts.buf_cache_size;

	/* buf_cache_size of UINT32_MAX means the value should be calculated dynamically

	 * based on the number of buffers in the shared pool and the number of poll groups

	 * that are sharing them.  We allocate 75% of the pool for the cache, and then

	 * divide that by number of poll groups to determine the buf_cache_size for this

	 * poll group.

	 */

	if (tgroup->buf_cache_size == UINT32_MAX) {

		uint32_t num_shared_buffers = transport->opts.num_shared_buffers;

		/* Theoretically the nvmf library can dynamically add poll groups to

		 * the target, after transports have already been created.  We aren't

		 * going to try to really handle this case efficiently, just do enough

		 * here to ensure we don't divide-by-zero.

		 */

		uint16_t num_poll_groups = group->tgt->num_poll_groups ? : spdk_env_get_core_count();



		tgroup->buf_cache_size = (num_shared_buffers * 3 / 4) / num_poll_groups;

	}



	bufs = calloc(tgroup->buf_cache_size, sizeof(struct spdk_nvmf_transport_pg_cache_buf *));



	if (!bufs) {