Commit 504bf591 authored by peluse's avatar peluse Committed by Tomasz Zawadzki
Browse files

lib/idxd: only select idxd device that are on the same socket 



Prior a regular round robin could result in strange performance
if an idxd device from another socket was used.

Signed-off-by: default avatarpeluse <peluse@localhost.localdomain>
Signed-off-by: default avatarZiye Yang <ziye.yang@intel.com>
Change-Id: Id863c79067beabe73ef89d92b3fb3c436821b97a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9367


Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: default avatarMonica Kenguva <monica.kenguva@intel.com>
Reviewed-by: default avatarJim Harris <james.r.harris@intel.com>
Reviewed-by: default avatarTomasz Zawadzki <tomasz.zawadzki@intel.com>
parent 17199cdc

CHANGELOG.md

+5 −0
Original line number Diff line number Diff line
@@ -2,6 +2,11 @@ 


## v21.10: (Upcoming Release)



### idxd



Added `spdk_idxd_get_socket` to query the socket that the idxd device

is on.



### nvmf



Added `oncs` to `struct spdk_nvmf_ctrlr_data` so that the transport layer

doc/accel_fw.md

+9 −0
Original line number Diff line number Diff line
@@ -86,6 +86,15 @@ of service parameters on the work queues that are not currently utilized by 
the module. Specialized use of DSA may require different configurations that

can be added to the module as needed.



When a new channel starts, a DSA device will be assigned to the channel. The accel

idxd module has been tuned for the most likely best performance case. The result

is that there is a limited number of channels that can be supported based on the

number of DSA devices in the system.  Additionally, for best performance, the accel

idxd module will only use DSA devices on the same socket as the requesting

channel/thread.  If an error occurs on initialization indicating that there are no

more DSA devices available either try fewer threads or, if on a 2 socket system,

try spreading threads across cores if possible.



### Software Module {#accel_sw}



The software module is enabled by default. If no hardware engine is explicitly

include/spdk/idxd.h

+8 −0
Original line number Diff line number Diff line
@@ -61,6 +61,14 @@ struct spdk_idxd_device; 
 */

struct idxd_batch;



/**

 * Get the socket that this device is on

 *

 * \param idxd device to query

 * \return socket number.

 */

uint32_t spdk_idxd_get_socket(struct spdk_idxd_device *idxd);



/**

 * Signature for configuring a channel

 *

lib/idxd/idxd.c

+6 −0
Original line number Diff line number Diff line
@@ -81,6 +81,12 @@ struct device_config g_dev_cfg1 = { 
	.total_engines = 4,

};



uint32_t

spdk_idxd_get_socket(struct spdk_idxd_device *idxd)

{

	return idxd->socket_id;

}



static inline void

_submit_to_hw(struct spdk_idxd_io_channel *chan, struct idxd_ops *op)

{

lib/idxd/idxd.h

+1 −0
Original line number Diff line number Diff line
@@ -168,6 +168,7 @@ struct spdk_idxd_impl { 
struct spdk_idxd_device {

	struct spdk_idxd_impl		*impl;

	void				*portals;

	uint32_t                        socket_id;

	int				wq_id;

	uint32_t			num_channels;

	uint32_t			total_wq_size;