Commit 835494b5 authored by Krzysztof Karas's avatar Krzysztof Karas Committed by Jim Harris
Browse files

docs: update scheduler documentation 



Updated scheduler documentation with information
about thread and core workloads, scheduling process
and dynamic scheduler parameters.

Added descriptions of new dynamic scheduler parameters
in jsonrpc.md.

Change-Id: I567efe21c425c13c16235d7976aad6ae961381bb
Signed-off-by: default avatarKrzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11886


Reviewed-by: default avatarTomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: default avatarJim Harris <james.r.harris@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
parent 8ec34933

doc/jsonrpc.md

+3 −0
Original line number Diff line number Diff line
@@ -713,6 +713,9 @@ Name | Optional | Type | Description 
----------------------- | -------- | ----------- | -----------

name                    | Required | string      | Name of a scheduler

period                  | Optional | number      | Scheduler period

load_limit              | Optional | number      | Thread load limit in % (dynamic only)

core_limit              | Optional | number      | Load limit on the core to be considered full (dynamic only)

core_busy               | Optional | number      | Indicates at what load on core scheduler should move threads to a different core (dynamic only)



#### Response

doc/scheduler.md

+28 −5
Original line number Diff line number Diff line
@@ -48,10 +48,11 @@ core has to be managed by the administrator. 
## Scheduler implementations



The scheduler in use may be controlled by JSON-RPC. Please use the

[framework_set_scheduler](jsonrpc.md/#rpc_framework_set_scheduler) RPC to

switch between schedulers or change their options.

[framework_set_scheduler](jsonrpc.html#rpc_framework_set_scheduler) RPC to

switch between schedulers or change their options. Currently only dynamic

scheduler supports changing its parameters.



[spdk_top](spdk_top.md#spdk_top) is a useful tool to observe the behavior of

[spdk_top](spdk_top.html#spdk_top) is a useful tool to observe the behavior of

schedulers in different scenarios and workloads.



### static [default]
@@ -65,11 +66,27 @@ the previous behavior of the SPDK event/application framework. 


The `dynamic` scheduler is designed for power saving and reduction of CPU

utilization, especially in cases where workloads show large variations over

time.

time. In SPDK thread and core workloads are measured in CPU ticks. Those

values are then compared with all the ticks since the last check, which allows

to calculate `busy time`.



`busy time = busy ticks / (busy tick + idle tick) * 100 %`



The thread is considered to be active, if its busy time is over the `load limit`

parameter.



Active threads are distributed equally among reactors, taking cpu_mask into

account. All idle threads are moved to the main core. Once an idle thread becomes

active, it is redistributed again.

active, it is redistributed again. Dynamic scheduler monitors core workloads and

redistributes SPDK threads on cores in a way that none of them is over `core limit`.

In case a core utilization surpasses this threshold, scheduler should move threads

out of it until this condition no longer applies. Cores might also be in overloaded

state, which indicates that moving threads out of this core will not decrease its

utilization under the `core limit` and the threads are unable to process all the I/O

they are capable of, because they share CPU ticks with other threads. The threshold

to decide if a core is overloaded is called `core busy`. Note that threads residing

on an overloaded core will not perform as good as other threads, because the CPU ticks

intended for them are limited by other threads on the same core.



When a reactor has no scheduled `spdk_thread`s it is switched into interrupt

mode and stops actively polling. After enough threads become active, the
@@ -80,3 +97,9 @@ not exceed the sum of all idle threads. When no active threads are present on 
the main core, the frequency of that CPU core will decrease as the load

decreases. All CPU cores corresponding to the other reactors remain at maximum

frequency.



The dynamic scheduler is currently the only one that allows manual setting of

its parameters.



Current values of scheduler parameters can be displayed by using

[framework_get_scheduler](jsonrpc.html#rpc_framework_get_scheduler) RPC.