Commit e8ea27f8 authored by Michal Berger's avatar Michal Berger Committed by Tomasz Zawadzki
Browse files

docker: Add docker-compose for building basic SPDK containers 



This suite can be used to deploy containers with the following
functionality (more details in README.md):

- storage-target
- proxy-container
- traffic-generator

This will run simple fio test as per fio.conf against nvmf controller
provided by initiator-container. Similar task can be performed directly
from initiator-container as well.

Each container includes SPDK installation with most common tools, e.g.
rpc.py, available under $PATH. This allows for something like:

docker-compose exec storage-target rpc.py nvmf_get_subsystems

Note that SPDK environment heavily depends on a running kernel hence all
the containers need to be privileged. That said, to make sure containers
are not affecting the host too much, some tasks must be done prior running
them. This includes:

- loading proper kernel modules (like nvme-fabrics, etc.)
- allocating hugepages and having at least one hugetlbfs mount
available under /dev/hugepages

base_build is created as docker multi-stage build.

This is done in order to decrease the size of the final image. The
SPDK RPMs are built inside a base image and then copied over to the
main image (+ fio binary) - this leaves all the dependencies inside
the intermediate image instead of the final one.

The resulted difference in size may look similar to the following
(it may differ depending on the docker version etc.):

no multi-stage build: spdk_base == 1.04GB
multi-stage build: spdk_base == 261MB

Signed-off-by: default avatarMichal Berger <michalx.berger@intel.com>
Signed-off-by: default avatarTomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I825bd0d0bb4071bd9d44b6a0749c033894899ae0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9055


Reviewed-by: default avatarJim Harris <james.r.harris@intel.com>
Reviewed-by: default avatarMonica Kenguva <monica.kenguva@intel.com>
Reviewed-by: default avatarXiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: default avatarChangpeng Liu <changpeng.liu@intel.com>
Tested-by: default avatarSPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
parent b098640f

docker/README.md

0 → 100644
+94 −0
Original line number Diff line number Diff line 
# SPDK Docker suite



This suite is meant to serve as an example of how SPDK can be encapsulated

into docker container images. The example containers consist of SPDK NVMe-oF

target sharing devices to another SPDK NVMe-oF application. Which serves

as both initiator and target. Finally a traffic generator based on FIO

issues I/O to the connected devices.



## Prerequisites



docker: We recommend version 20.10 and above because it supports cgroups v2 for

customization of host resources like CPUs, memory, and block I/O.



docker-compose: We recommend using 1.29.2 version or newer.



kernel: Hugepages must be allocated prior running the containers and hugetlbfs

mount must be available under /dev/hugepages. Also, tmpfs should be mounted

under /dev/shm. Depending on the use-case, some kernel modules should be also

loaded into the kernel prior running the containers.



proxy: If you are working behind firewall make sure dockerd is aware of the

proxy. Please refer to:

[docker-proxy](https://docs.docker.com/config/daemon/systemd/#httphttps-proxy)



To pass `$http_proxy` to docker-compose build use:

~~~{.sh}

docker-compose build --build-arg PROXY=$http_proxy

~~~



## How-To



`docker-compose.yaml` shows an example deployment of the storage containers based on SPDK.

Running `docker-compose build` creates 4 docker images:



- build_base

- storage-target

- proxy-container

- traffic-generator



The `build_base` image provides the core components required to containerize SPDK

applications. The fedora:33 image from the Fedora Container Registry is used and then SPDK is installed. SPDK is installed out of `build_base/spdk.tar.gz` provided.

See `build_base` folder for details on what's included in the final image.



Running `docker-compose up` creates 3 docker containers:



-- storage-target: Contains SPDK NVMe-oF target exposing single subsystem to

`proxy-container` based on malloc bdev.

-- proxy-container: Contains SPDK NVMe-oF target connecting to `storage-target`

and then exposing the same subsystem to `traffic-generator`.

-- traffic-generator: Contains FIO using SPDK plugin to connect to `proxy-container`

and runs a sample workload.



Each container is connected to a separate "spdk" network which is created before

deploying the containers. See `docker-compose.yaml` for the network's detailed setup and ip assignment.



All the above boils down to:



~~~{.sh}

cd docker

tar -czf build_base/spdk.tar.gz --exclude='docker/*' -C .. .

docker-compose build

docker-compose up

~~~



The `storage-target` and `proxy-container` can be started as services.

Allowing for multiple `traffic-generator` containers to connect.



~~~{.sh}

docker-compose up -d proxy-container

docker-compose run traffic-generator

~~~



Enviroment variables to containers can be passed as shown in

[docs](https://docs.docker.com/compose/environment-variables/).

For example extra arguments to fio can be passed as so:



~~~{.sh}

docker-compose run -e FIO_ARGS="--minimal" traffic-generator

~~~



As each container includes SPDK installation it is possible to use rpc.py to

examine the final setup. E.g.:



~~~{.sh}

docker-compose exec storage-target rpc.py bdev_get_bdevs

docker-compose exec proxy-container rpc.py nvmf_get_subsystems

~~~



## Caveats



- If you run docker < 20.10 under distro which switched fully to cgroups2

  (e.g. f33) make sure that /sys/fs/cgroup/systemd exists otherwise docker/build

  will simply fail.

- Each SPDK app inside the containers is limited to single, separate CPU.

docker/build_base/Dockerfile

0 → 100644
+40 −0
Original line number Diff line number Diff line 
# SPDX-License-Identifier: Apache-2.0

# Copyright (c) Intel Corporation



FROM fedora:33 AS base



# Generic args

ARG PROXY

ARG NO_PROXY



ENV http_proxy=$PROXY

ENV https_proxy=$PROXY

ENV no_proxy=$NO_PROXY





COPY spdk.tar.gz /spdk.tar.gz

COPY pre-install /install

RUN /install



# We are doing a multi-stage build here. This means that previous image,

# base, is going to end up as an intermediate one, untagged, <none> - this

# image can be then manually removed (--force-rm doesn't work here. Go

# figure).

FROM fedora:33 AS spdk



LABEL maintainer=spdk.io



# Proxy configuration must be set for each build separately...

ARG PROXY

ARG NO_PROXY



ENV http_proxy=$PROXY

ENV https_proxy=$PROXY

ENV no_proxy=$NO_PROXY



# Copy SPDK's RPMs built during pre-install step.

COPY --from=base /tmp/*.rpm /tmp/

COPY --from=base /tmp/fio /tmp/

# Wrap up the image

COPY post-install /install

RUN /install

docker/build_base/post-install

0 → 100755
+20 −0
Original line number Diff line number Diff line 
#!/usr/bin/env bash

set -e



dnf install -y /tmp/*.rpm



# Be nice for docker exec and link SPDK scripts|binaries under common PATH

# location like /usr/bin.

ln -sf $(ls -1dpA /usr/local/bin/* | grep -v "/$") /usr/bin

ln -sf $(ls -1dpA /usr/local/bin/fio/* | grep -v "/$") /usr/bin

ln -s /usr/libexec/spdk/scripts/rpc.py /usr/bin

ln -s /usr/libexec/spdk/scripts/rpc_http_proxy.py /usr/bin

ln -s /usr/libexec/spdk/scripts/setup.sh /usr/bin

ln -s /usr/libexec/spdk/include/spdk /usr/include

ln -s /usr/libexec/spdk/scripts/ /usr



mkdir -p /usr/src/fio

mv /tmp/fio /usr/src/fio



dnf clean all

rm -f /tmp/*.rpm

docker/build_base/pre-install

0 → 100755
+43 −0
Original line number Diff line number Diff line 
#!/usr/bin/env bash

set -e



spdk_repo=$(mktemp -dt "spdk.XXXXXX")

spdk_tar=/spdk.tar.gz



cleanup() {



	rm -f "$HOME/rpmbuild/rpm/x86_64/"*.rpm

	rm -f "$spdk_tar"

	rm -rf "$spdk_repo"

}



trap 'cleanup' EXIT



if [[ ! -e $spdk_tar ]]; then

	printf 'Missing %s\n' "$spdk_tar" >&2

	exit 1

fi



tar -C "$spdk_repo" -xf "$spdk_tar"



# Required for building RPM

dnf install -y rpm-build



# Spice it a bit with supported sources

"$spdk_repo/scripts/pkgdep.sh" -d

"$spdk_repo/test/common/config/vm_setup.sh" --test-conf=fio



# HACK: In case we received a .tar with built SPDK we need to overwrite the

# configuration to update all the paths make would need to lookup - this is

# needed since we execute inside a different mount namespace so we won't be

# able to find any absoulte paths that were used prior creating the .tar.

"$spdk_repo/configure" > /dev/null



# Deploy SPDK inside the container

DEPS="no" "$spdk_repo/rpmbuild/rpm.sh" \

	--with-shared \

	--with-fio



mv "$HOME/rpmbuild/rpm/x86_64/"*.rpm /tmp

mv "/usr/src/fio/fio" /tmp

dnf clean all

docker/docker-compose.yaml

0 → 100644
+62 −0
Original line number Diff line number Diff line 
# SPDX-License-Identifier: Apache-2.0

# Copyright (c) Intel Corporation

version: "3.8"

services:

  build_base:

    image: spdk

    build:

      context: build_base

    container_name: build_base

  storage-target:

    image: spdk-app

    build:

      context: spdk-app

    container_name: storage-target

    depends_on:

      - build_base

    networks:

      spdk:

        ipv4_address: 192.168.42.2

    volumes:

      - /dev/hugepages:/dev/hugepages

      - ./spdk-app/storage-target.conf:/config

    environment:

      - SPDK_ARGS=-m 0x2

    privileged: true

  proxy-container:

    image: spdk-app

    build:

      context: spdk-app

    container_name: proxy-container

    depends_on:

      - storage-target

    networks:

      spdk:

        ipv4_address: 192.168.42.3

    volumes:

      - /dev/hugepages:/dev/hugepages

      - ./spdk-app/proxy-container.conf:/config

    environment:

      - SPDK_ARGS=-m 0x4

    privileged: true

  traffic-generator:

    image: traffic-generator

    build:

      context: traffic-generator

    container_name: traffic-generator

    depends_on:

      - proxy-container

    networks:

      spdk:

    volumes:

      - /dev/hugepages:/dev/hugepages

      - ./traffic-generator/conf:/config

      - ./traffic-generator/fio.conf:/fio.conf

    privileged: true

networks:

  spdk:

    name: "spdk"

    ipam:

      config:

        - subnet: 192.168.42.0/29

          gateway: 192.168.42.1