Commit 8753fdb6 authored May 20, 2020 by Shuhei Matsumoto Committed by Tomasz Zawadzki May 22, 2020

example/nvmf: Run SPDK threads by FIFO to avoid deadlock



if spdk_thread_poll() calls spdk_thread_create() to create SPDK
thread dynamically, it will acquire nvmf_reactor->mutex and will
cause deadlock because the queue nvmf_reactor->threads is accessed
by any POSIX thread by holding nvmf_reactor->mutex.

If any non-owner POSIX thread manipulate nvmf_reactor->threads by
holding nvmf_reactor->threads, like general priority based scheduler,
running SPDK threads by FIFO will be better.

Change nvmf_reactor_run() to dequeue a SPDK thread from the head, run
it, and then enqueue it to the tail. Release nvmf_reactor->mutex
while spdk_thread_poll() is executed.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I9b4040cc4d995beaf4a1f9ed8260bb1254d7d7fb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2537


Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

parent 9bea56c0

examples/nvmf/nvmf/nvmf.c

+9 −2

Original line number	Diff line number	Diff line
		@@ -186,13 +186,20 @@ nvmf_reactor_run(void *arg)
		struct nvmf_lw_thread lw_thread, tmp;
		struct spdk_thread *thread;

		/* foreach all the lightweight threads in this nvmf_reactor */
		/* run all the lightweight threads in this nvmf_reactor by FIFO. */
		do {
		pthread_mutex_lock(&nvmf_reactor->mutex);
		TAILQ_FOREACH_SAFE(lw_thread, &nvmf_reactor->threads, link, tmp) {
		lw_thread = TAILQ_FIRST(&nvmf_reactor->threads);
		if (lw_thread != NULL) {
		TAILQ_REMOVE(&nvmf_reactor->threads, lw_thread, link);
		pthread_mutex_unlock(&nvmf_reactor->mutex);

		thread = spdk_thread_get_from_ctx(lw_thread);

		spdk_thread_poll(thread, 0, 0);

		pthread_mutex_lock(&nvmf_reactor->mutex);
		TAILQ_INSERT_TAIL(&nvmf_reactor->threads, lw_thread, link);
		}
		pthread_mutex_unlock(&nvmf_reactor->mutex);
		} while (!g_reactors_exit);

Admin message