Commit 338475bd authored Jun 21, 2024 by Alex Michon Committed by Tomasz Zawadzki Jul 01, 2024

lib/thread: Align spdk_thread allocation on cache line



spdk_thread is a special object in SPDK because it is not allocated on
the core that is going to use it.
Consecutive mallocs on the same core may share the same cache line. So
it is possible that a spdk_thread object shares a cache line with any
other object. The problem is that the spdk_thread object will be used by
the core actually running the thread and the 2nd object will likely be
used by the core that created the spdk_thread. This is bad for
performance and it is even worse if the two cores are not in the same
socket.

Change-Id: I0628b64ee261901925b7005ba8daee235fdbd592
Signed-off-by: Alex Michon <amichon@kalrayinc.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/23814


Reviewed-by: Jim Harris <jim.harris@samsung.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

parent cde2142c

lib/thread/thread.c

+6 −2

Original line number	Diff line number	Diff line
		@@ -490,14 +490,18 @@ struct spdk_thread *
		spdk_thread_create(const char name, const struct spdk_cpuset cpumask)
		{
		struct spdk_thread thread, null_thread;
		size_t size = SPDK_ALIGN_CEIL(sizeof(*thread) + g_ctx_sz, SPDK_CACHE_LINE_SIZE);
		struct spdk_msg *msgs[SPDK_MSG_MEMPOOL_CACHE_SIZE];
		int rc = 0, i;

		thread = calloc(1, sizeof(*thread) + g_ctx_sz);
		if (!thread) {
		/* Since this spdk_thread object will be used by another core, ensure that it won't share a
		* cache line with any other object allocated on this core */
		rc = posix_memalign((void **)&thread, SPDK_CACHE_LINE_SIZE, size);
		if (rc != 0) {
		SPDK_ERRLOG("Unable to allocate memory for thread\n");
		return NULL;
		}
		memset(thread, 0, size);

		if (cpumask) {
		spdk_cpuset_copy(&thread->cpumask, cpumask);