Commit 8e85b675 authored by Alexey Marchuk's avatar Alexey Marchuk Committed by Tomasz Zawadzki
Browse files

sock: Add new params to configure zcopy for server, client sockets 



When zcero copy send is enabled and used by initiator,
it could significantly increase latency in some payloads.
To enable more fine graing configuration of zero copy
send feature, add new parameters enable_zerocopy_send_server
and enable_zerocopy_send_client to spdk_sock_impl_opts to
enable/disable zcopy for specific type of sockets.
Exisiting enable_zerocopy_send parameter affects all types
of sockets.

Signed-off-by: default avatarAlexey Marchuk <alexeymar@mellanox.com>
Change-Id: I111c75608f8826980a56e210c076ab8ff16ddbdc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7457


Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: default avatarTomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: default avatarKarol Latecki <karol.latecki@intel.com>
Reviewed-by: default avatarShuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
parent 0859db6b

CHANGELOG.md

+6 −0
Original line number Diff line number Diff line
@@ -179,6 +179,12 @@ sockets to be marked using the SO_MARK socket option as a hint for which hardwar 
queue they should be associated with. This mode leverages that by setting the same

value for all sockets within a poll group.



New parameters `enable_zerocopy_send_server` and `enable_zerocopy_send_client` were added

to struct spdk_sock_impl_opts, these parameters enable or disable zero copy send for server

and client sockets which are created using `spdk_sock_listen` and `spdk_sock_listen_ext` (server);

`spdk_sock_connect` and `spdk_sock_connect_ext` (client) functions. Existing parameter

`enable_zerocopy_send`  enables or disables zero copy send for both server and client sockets.



### thread



A new API `spdk_io_channel_get_io_device` was added to get the io_device for the specified

doc/jsonrpc.md

+19 −11
Original line number Diff line number Diff line
@@ -8591,7 +8591,11 @@ Example response: 
    "recv_buf_size": 2097152,

    "send_buf_size": 2097152,

    "enable_recv_pipe": true,

    "enable_zerocopy_send": true

    "enable_zerocopy_send": true,

    "enable_quickack": true,

    "enable_placement_id": 0,

    "enable_zerocopy_send_server": true,

    "enable_zerocopy_send_client": false

  }

}

~~~
@@ -8603,14 +8607,16 @@ Set parameters for the socket layer implementation. 
### Parameters



Name                        | Optional | Type        | Description

----------------------- | -------- | ----------- | -----------

--------------------------- | -------- | ----------- | -----------

impl_name                   | Required | string      | Name of socket implementation, e.g. posix

recv_buf_size               | Optional | number      | Size of socket receive buffer in bytes

send_buf_size               | Optional | number      | Size of socket send buffer in bytes

enable_recv_pipe            | Optional | boolean     | Enable or disable receive pipe

enable_zerocopy_send    | Optional | boolean     | Enable or disable zero copy on send

enable_zerocopy_send        | Optional | boolean     | Enable or disable zero copy on send for client and server sockets

enable_quick_ack            | Optional | boolean     | Enable or disable quick ACK

enable_placement_id         | Optional | number      | Enable or disable placement_id. 0:disable,1:incoming_napi,2:incoming_cpu

enable_zerocopy_send_server | Optional | boolean     | Enable or disable zero copy on send for server sockets

enable_zerocopy_send_client | Optional | boolean     | Enable or disable zero copy on send for client sockets



### Response
@@ -8632,7 +8638,9 @@ Example request: 
    "enable_recv_pipe": false,

    "enable_zerocopy_send": true,

    "enable_quick_ack": false,

    "enable_placement_id": 0

    "enable_placement_id": 0,

    "enable_zerocopy_send_server": true,

    "enable_zerocopy_send_client": false

  }

}

~~~

examples/nvme/perf/perf.c

+1 −0
Original line number Diff line number Diff line
@@ -335,6 +335,7 @@ perf_set_sock_zcopy(const char *impl_name, bool enable) 
	}



	sock_opts.enable_zerocopy_send = enable;

	sock_opts.enable_zerocopy_send_client = enable;



	if (spdk_sock_impl_set_opts(impl_name, &sock_opts, opts_size)) {

		fprintf(stderr, "Failed to %s zcopy send for sock impl %s: error %d (%s)\n",

include/spdk/sock.h

+9 −0
Original line number Diff line number Diff line
@@ -126,6 +126,15 @@ struct spdk_sock_impl_opts { 
	 */

	uint32_t enable_placement_id;



	/**

	 * Enable or disable use of zero copy flow on send for server sockets. Used by posix socket module.

	 */

	bool enable_zerocopy_send_server;



	/**

	 * Enable or disable use of zero copy flow on send for client sockets. Used by posix socket module.

	 */

	bool enable_zerocopy_send_client;

};



/**

lib/sock/sock.c

+2 −0
Original line number Diff line number Diff line
@@ -791,6 +791,8 @@ spdk_sock_write_config_json(struct spdk_json_write_ctx *w) 
			spdk_json_write_named_bool(w, "enable_zerocopy_send", opts.enable_zerocopy_send);

			spdk_json_write_named_bool(w, "enable_quickack", opts.enable_quickack);

			spdk_json_write_named_uint32(w, "enable_placement_id", opts.enable_placement_id);

			spdk_json_write_named_bool(w, "enable_zerocopy_send_server", opts.enable_zerocopy_send_server);

			spdk_json_write_named_bool(w, "enable_zerocopy_send_client", opts.enable_zerocopy_send_client);

			spdk_json_write_object_end(w);

			spdk_json_write_object_end(w);

		} else {