bdev: add spdk_bdev_io padding in place of io_submit_ch

Commit 055de83a ("bdev: multiple QoS queues with atomic-based QoS
quota") changed the way bdev layer handles IOs with QoS enabled.  It
shouldn't affect IOs when QoS is disabled.  However, apparently, this
change *has* shown performance drop on some workloads.  Specifically,
single core bdepverf performance running on 7 Kioxia KCM61VUL3T20 drives
with qd=192 went down from ~5.9M to ~5.4M.

As part of that commit, the layout of struct spdk_bdev_io has been
changed due to removing io_submit_ch.  Re-adding the padding in place of
that field has proven to improve IOPS in the case mentioned above from
~5.3M (current master) to ~5.7M.  This patch does exactly that, but we
should look into the layout of this structure more thoroughly in the
future.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I35a04b93bec54620a6a2c8f1133870f5f101eda9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/21576


Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>