+11
−0
+46
−1
Loading
lib/thread: Add API to register a post poller handler
This patch aims to improve doorbell management. SPDK pollers are executed one after other, there is no way to controll its execution time. In a case when some application has 3 active pollers where each poller represents a HW queue (e.g. nvmf_tgt with nvme_tcp backend and accel module), time interval between 2 poller invokations might be much more than average request latency. That makes doorbells managment more complicated bcs the rule of thumb is to submit a batch of commands to a HW and typical design is to ring doorbells once per poller invokation to flush any tasks submitted by other pollers. With this approach (flush once per poll) HW stalls for significant amount of time. The best solution would be to flush requests to the HW once a batch of requests is in the submission queue. But that is another poller which submits work to the queue, e.g. nvmf_tgt reaps a new request and submits it do bdev and bdev submits crypto task. That is done in the context of nvmf_tgt poller but task is eventually submitted to HW in the context of accel poller. We need a mechanism to notify accel module that nvmf_tgt finishes a batch in order to flush submission queue. But we can go in another direction - code in accel module may ask nvmf_tgt poller to flush doorbells once it submits the last request in the batch. With some abstraction we can achieve it by introducing generic API which registers a one-shot callback which a poller calls once it exits its routine task. In that case accel module can register a callback to flush SQ if a new request is submitted. This patch adds such API It allows to ring doorbels (and in the future to flush sockets) in more optimal way and significantly improve performance in highly loaded applications. Signed-off-by:Alexey Marchuk <alexeymar@nvidia.com> Change-Id: I026cf931f6e56bfbb66422738a90f71525516bf7 Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/24703 Reviewed-by: