doc: Integrate concurrency and event framework docs

infrastructure required to call spdk_allocate_thread(). This includes spawning

one thread per core, pinning each thread to a unique core, and allocating

lockless rings between the threads for message passing. Instead of

re-implementing that infrastructure for each example application, SPDK provides

the SPDK [event framework](http://www.spdk.io/doc/event_8h.html). This library

handles setting up all of the message passing infrastructure, installing signal

handlers to cleanly shutdown, implements periodic pollers, and does basic

command line parsing. When started through spdk_app_start(), the library

automatically spawns all of the threads requested, pins them, and calls

spdk_allocate_thread() with appropriate function pointers for each one. This

makes it much easier to implement a brand new SPDK application and is the

recommended method for those starting out. Only established applications with

sufficient message passing infrastructure should consider directly integrating

the lower level libraries.

re-implementing that infrastructure for each example application, SPDK

provides the SPDK @ref event. This library handles setting up all of the

message passing infrastructure, installing signal handlers to cleanly

shutdown, implements periodic pollers, and does basic command line parsing.

When started through spdk_app_start(), the library automatically spawns all of

the threads requested, pins them, and calls spdk_allocate_thread() with

appropriate function pointers for each one. This makes it much easier to

implement a brand new SPDK application and is the recommended method for those

starting out. Only established applications with sufficient message passing

infrastructure should consider directly integrating the lower level libraries.

# Limitations of the C Language

# Event framework {#event}

# Event Framework {#event}

SPDK provides a framework for writing asynchronous, polled-mode, shared-nothing server applications.

The event framework is intended to be optional; most other SPDK components are designed to be

integrated into an application without specifically depending on the SPDK event library.

The framework defines several concepts - reactors, events, and pollers - that are described

in the following sections.

The event framework spawns one thread per core (reactor) and connects the threads with

lockless queues.

Messages (events) can then be passed between the threads.

On modern CPU architectures, message passing is often much faster than traditional locking.

SPDK provides a framework for writing asynchronous, polled-mode,

shared-nothing server applications. The event framework is intended to be

optional; most other SPDK components are designed to be integrated into an

application without specifically depending on the SPDK event library. The

framework defines several concepts - reactors, events, and pollers - that are

described in the following sections. The event framework spawns one thread per

core (reactor) and connects the threads with lockless queues. Messages

(events) can then be passed between the threads. On modern CPU architectures,

message passing is often much faster than traditional locking. For a

discussion of the theoretical underpinnings of this framework, see @ref

concurrency.

The event framework public interface is defined in spdk/event.h.

The event framework public interface is defined in event.h.

# Event Framework Design Considerations {#event_design}

Simple server applications can be written in a single-threaded fashion. This allows for

straightforward code that can maintain state without any locking or other synchronization.

However, to scale up (for example, to allow more simultaneous connections), the application may

need to use multiple threads.

In the ideal case where each connection is independent from all other connections,

the application can be scaled by creating additional threads and assigning connections to them

without introducing cross-thread synchronization.

Unfortunately, in many real-world cases, the connections are not entirely independent

and cross-thread shared state is necessary.

SPDK provides an event framework to help solve this problem.

Simple server applications can be written in a single-threaded fashion. This

allows for straightforward code that can maintain state without any locking or

other synchronization. However, to scale up (for example, to allow more

simultaneous connections), the application may need to use multiple threads.

In the ideal case where each connection is independent from all other

connections, the application can be scaled by creating additional threads and

assigning connections to them without introducing cross-thread

synchronization. Unfortunately, in many real-world cases, the connections are

not entirely independent and cross-thread shared state is necessary. SPDK

provides an event framework to help solve this problem.

# SPDK Event Framework Components {#event_components}

## Events {#event_component_events}

To accomplish cross-thread communication while minimizing synchronization overhead,

the framework provides message passing in the form of events.

The event framework runs one event loop thread per CPU core.

These threads are called reactors, and their main responsibility is to process incoming events

from a queue.

Each event consists of a bundled function pointer and its arguments, destined for

a particular CPU core.

Events are created using spdk_event_allocate() and executed using spdk_event_call().

Unlike a thread-per-connection server design, which achieves concurrency by depending on the

operating system to schedule many threads issuing blocking I/O onto a limited number of cores,

the event-driven model requires use of explicitly asynchronous operations to achieve concurrency.

Asynchronous I/O may be issued with a non-blocking function call, and completion is typically

signaled using a callback function.

To accomplish cross-thread communication while minimizing synchronization

overhead, the framework provides message passing in the form of events. The

event framework runs one event loop thread per CPU core. These threads are

called reactors, and their main responsibility is to process incoming events

from a queue. Each event consists of a bundled function pointer and its

arguments, destined for a particular CPU core. Events are created using

spdk_event_allocate() and executed using spdk_event_call(). Unlike a

thread-per-connection server design, which achieves concurrency by depending

on the operating system to schedule many threads issuing blocking I/O onto a

limited number of cores, the event-driven model requires use of explicitly

asynchronous operations to achieve concurrency. Asynchronous I/O may be issued

with a non-blocking function call, and completion is typically signaled using

a callback function.

## Reactors {#event_component_reactors}

Each reactor has a lock-free queue for incoming events to that core, and threads from any core

may insert events into the queue of any other core.

The reactor loop running on each core checks for incoming events and executes them in

first-in, first-out order as they are received.

Event functions should never block and should preferably execute very quickly,

since they are called directly from the event loop on the destination core.

Each reactor has a lock-free queue for incoming events to that core, and

threads from any core may insert events into the queue of any other core. The

reactor loop running on each core checks for incoming events and executes them

in first-in, first-out order as they are received. Event functions should

never block and should preferably execute very quickly, since they are called

directly from the event loop on the destination core.

## Pollers {#event_component_pollers}

The framework also defines another type of function called a poller.

Pollers may be registered with the spdk_poller_register() function.

Pollers, like events, are functions with arguments that can be bundled and sent to a specific

core to be executed.

However, unlike events, pollers are executed repeatedly until unregistered.

The reactor event loop intersperses calls to the pollers with other event processing.

Pollers are intended to poll hardware as a replacement for interrupts.

Normally, pollers are executed on every iteration of the main event loop.

Pollers may also be scheduled to execute periodically on a timer if low latency is not required.

The framework also defines another type of function called a poller. Pollers

may be registered with the spdk_poller_register() function. Pollers, like

events, are functions with arguments that can be bundled and executed.

However, unlike events, pollers are executed repeatedly until unregistered and

are executed on the thread they are registered on. The reactor event loop

intersperses calls to the pollers with other event processing. Pollers are

intended to poll hardware as a replacement for interrupts. Normally, pollers

are executed on every iteration of the main event loop. Pollers may also be

scheduled to execute periodically on a timer if low latency is not required.

## Application Framework {#event_component_app}

- @ref bdev_module

- @ref directory_structure

- [Public API header files](files.html)

- @ref event

# Modules {#modules}

- @ref event

- @ref nvme

- @ref nvmf

- @ref ioat