Unverified Commit 3e465431 authored by John DiSanti's avatar John DiSanti Committed by GitHub
Browse files

Simplify CI with composite actions and new image tagging approach (#1283) 

- Refactors CI to use composite actions so that there's less duplication in the workflow yaml files.
- Switches to a different image tagging approach for the Docker build image: now the git hash of the `tools/` directory is the image tag rather than the commit hash of `HEAD`. This will result in more image reuse if the tools haven't changed, and makes the logic for deciding whether a new image must be built much simpler.
- When running CI on main, ensures the Docker build image is built and uploaded to ECR before running the CI tests.
parent 6fb5acc9

.github/actions/docker-build/action.yml

0 → 100644
+64 −0
Original line number Diff line number Diff line 
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

# SPDX-License-Identifier: Apache-2.0.



# Use this action to execute the action scripts in tools/ci-build/scripts within the Docker build image.

name: smithy-rs Docker Build

description: Run Docker build command for smithy-rs

inputs:

  # The name of the script in tools/ci-build/scripts to run

  action:

    description: What action to run in the Docker build

    required: true

runs:

  using: composite

  steps:

  - uses: actions/cache@v2

    name: Gradle Cache

    with:

      path: |

        gradle/caches

        gradle/wrapper

      key: ${{ runner.os }}-gradle-${{ hashFiles('gradle/caches/**/*', 'gradle/wrapper/**/*') }}

      restore-keys: |

        ${{ runner.os }}-gradle-

      # Pinned to the commit hash of v1.3.0

  - uses: Swatinem/rust-cache@842ef286fff290e445b90b4002cc9807c3669641

    with:

      sharedKey: ${{ runner.os }}-${{ github.job }}

      target-dir: ./smithy-rs-target

  - name: Download all artifacts

    uses: ./smithy-rs/.github/actions/download-all-artifacts

  - name: Prepare build image

    shell: bash

    run: |

      set -x



      # Check the build artifacts to see if a prior step built a new Docker build image.

      # If smithy-rs-base-image was included in the downloaded build artifacts, then load

      # it and tag it as the base image to use for this action. This will prevent acquire-build-image

      # from attempting to download an image from ECR since it will already exist,

      # which enables testing build image modifications as part of the pull request.

      if [[ -d smithy-rs-base-image ]]; then

        IMAGE_TAG="$(./smithy-rs/tools/ci-build/tools-hash)"

        docker load -i smithy-rs-base-image/smithy-rs-base-image

        docker tag "smithy-rs-base-image:${IMAGE_TAG}" "smithy-rs-base-image:local"

      fi



      # For this step, we want images to come from build artifacts (built as part a prior step),

      # or from ECR. We disable building the image from scratch so that any mistakes in the CI

      # configuration won't cause each individual action to build its own image, which would

      # drastically increase the total CI time. Fail fast!

      ALLOW_LOCAL_BUILD=false ./smithy-rs/tools/ci-build/acquire-build-image

    # This runs the commands from the matrix strategy

  - name: Run ${{ inputs.action }}

    shell: bash

    run: |

      ./smithy-rs/tools/ci-build/ci-action ${{ inputs.action }}

      tar cfz artifacts-${{ inputs.action }}.tar.gz -C artifacts .

  - name: Upload artifacts

    uses: actions/upload-artifact@v3

    with:

      name: artifacts-${{ inputs.action }}

      path: artifacts-${{ inputs.action }}.tar.gz

      if-no-files-found: error

      retention-days: 3

.github/actions/download-all-artifacts/action.yml

0 → 100644
+13 −0
Original line number Diff line number Diff line 
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

# SPDX-License-Identifier: Apache-2.0.



name: Download All Artifacts

description: Downloads and untars all available build artifacts

runs:

  using: composite

  steps:

  - name: Download artifacts

    uses: actions/download-artifact@v3

  - name: Untar artifacts

    shell: bash

    run: find . -maxdepth 2 -iname 'artifacts-*.tar.gz' -print -exec tar vxfz {} \;

.github/workflows/ci-docker-build.yml→.github/workflows/ci-main.yml

+17 −6
Original line number Diff line number Diff line 
name: Docker Build Image

# This workflow differs from PR CI in that it uploads a Docker build image to public ECR.

# This should be done only on push to main so that PRs from forks can successfully run CI

# since GitHub secrets cannot be shared with a PR from a fork.

name: CI on Branch `main`

on:

  workflow_dispatch:

  push:

    branches: [main]

    paths:

    - tools/**



# Allow only one Docker build image build to run at a time for the entire smithy-rs repo

concurrency:
@@ -15,6 +16,7 @@ env: 
  ecr_repository: public.ecr.aws/w0m4q9l7/github-awslabs-smithy-rs-ci



jobs:

  # Rebuild and upload the Docker build image

  rebuild-docker-build-image:

    runs-on: ubuntu-latest

    name: Rebuild image
@@ -26,9 +28,12 @@ jobs: 
      uses: actions/checkout@v2

    - name: Build image

      run: |

        IMAGE_TAG="$(git rev-parse HEAD)"

        IMAGE_TAG="$(./tools/ci-build/tools-hash)"

        cd tools

        docker build -t "${{ env.ecr_repository }}:${IMAGE_TAG}" .

        docker build \

          -t "${{ env.ecr_repository }}:${IMAGE_TAG}" \

          -t "${{ env.ecr_repository }}:main" \

          .

    - name: Acquire credentials

      uses: aws-actions/configure-aws-credentials@v1

      with:
@@ -37,6 +42,12 @@ jobs: 
        aws-region: us-west-2

    - name: Upload image

      run: |

        IMAGE_TAG="$(git rev-parse HEAD)"

        IMAGE_TAG="$(./tools/ci-build/tools-hash)"

        aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

        docker push "${{ env.ecr_repository }}:${IMAGE_TAG}"

        docker push "${{ env.ecr_repository }}:main"



  # Run normal CI, which should pick up the updated build image

  ci:

    needs: rebuild-docker-build-image

    uses: awslabs/smithy-rs/.github/worfklows/ci.yaml@main

.github/workflows/ci.yaml

+25 −135
Original line number Diff line number Diff line 
on:

  push:

    branches: [main]

    tags:

    - '*'

  workflow_call:

  workflow_dispatch:

  pull_request:



name: CI
@@ -23,16 +21,14 @@ jobs: 
  acquire-base-image:

    name: Acquire Base Image

    runs-on: ubuntu-latest

    outputs:

      image-in-artifacts: ${{ steps.acquire.outputs.image-in-artifacts }}

    steps:

    - uses: actions/checkout@v2

    - uses: actions/checkout@v3

      with:

        path: smithy-rs

        fetch-depth: 0

    - name: Acquire base image

      id: acquire

      # Script sets boolean output value named `image-in-artifacts`

      run: ./smithy-rs/tools/ci-build/ci-output-build-image ${{ github.event.pull_request.base.sha }}

      run: ./smithy-rs/tools/ci-build/acquire-build-image

    - name: Upload base image

      uses: actions/upload-artifact@v3

      with:
@@ -58,52 +54,17 @@ jobs: 
        - action: generate-aws-sdk-smoketest

        - action: generate-smithy-rs-runtime-bundle

    steps:

    - uses: actions/checkout@v2

    - uses: actions/checkout@v3

      with:

        path: smithy-rs

    - uses: actions/checkout@v2

    - uses: actions/checkout@v3

      with:

        repository: awsdocs/aws-doc-sdk-examples

        path: aws-doc-sdk-examples

    - uses: actions/cache@v2

      name: Gradle Cache

      with:

        path: |

          gradle/caches

          gradle/wrapper

        key: ${{ runner.os }}-gradle-${{ hashFiles('gradle/caches/**/*', 'gradle/wrapper/**/*') }}

        restore-keys: |

          ${{ runner.os }}-gradle-

      # Pinned to the commit hash of v1.3.0

    - uses: Swatinem/rust-cache@842ef286fff290e445b90b4002cc9807c3669641

      with:

        sharedKey: ${{ runner.os }}-${{ env.rust_version }}-${{ github.job }}

        target-dir: ./smithy-rs-target

    - name: Download build image

      if: ${{ needs.acquire-base-image.outputs.image-in-artifacts == 'true' }}

      uses: actions/download-artifact@v3

      with:

        name: smithy-rs-base-image

    - name: Prepare build image

      run: |

        if [[ "${{ needs.acquire-base-image.outputs.image-in-artifacts }}" == "true" ]]; then

          docker load -i smithy-rs-base-image

        else

          ./smithy-rs/tools/ci-build/acquire-base-image --force-remote ${{ github.event.pull_request.base.sha }}

        fi

        ./smithy-rs/tools/ci-build/create-local-build-image

      # This runs the commands from the matrix strategy

    - name: Run ${{ matrix.actions.action }}

      run: |

        ./smithy-rs/tools/ci-build/ci-action ${{ matrix.actions.action }}

        tar cfz artifacts-${{ matrix.actions.action }}.tar.gz -C artifacts .

    - name: Upload artifacts

      uses: actions/upload-artifact@v3

      uses: ./smithy-rs/.github/actions/docker-build

      with:

        name: artifacts-${{ matrix.actions.action }}

        path: artifacts-${{ matrix.actions.action }}.tar.gz

        if-no-files-found: error

        retention-days: 3

        action: ${{ matrix.actions.action }}



  test:

    name: Test
@@ -115,68 +76,27 @@ jobs: 
    # in a matrix strategy. These commands get run in the steps after all the setup.

    strategy:

      fail-fast: false

      max-parallel: 7

      matrix:

        # These correspond to scripts in tools/ci-build/scripts that will be run in the Docker build image

        test:

        # Kick off the slowest three first

        - action: check-aws-sdk-services

        - action: check-client-codegen-unit-tests

        - action: check-rust-runtimes-and-tools

        # Order by fastest to slowest

        - action: check-server-codegen-unit-tests

        - action: check-server-codegen-integration-tests

        - action: check-sdk-codegen-unit-tests

        - action: check-client-codegen-integration-tests

        - action: check-aws-sdk-smoketest-additional-checks

        - action: check-aws-sdk-smoketest-docs-clippy-udeps

        - action: check-aws-sdk-smoketest-unit-tests

        - action: check-client-codegen-integration-tests

        - action: check-client-codegen-unit-tests

        - action: check-rust-runtimes-and-tools

        - action: check-sdk-codegen-unit-tests

        - action: check-server-codegen-integration-tests

        - action: check-server-codegen-unit-tests

    steps:

    - uses: actions/checkout@v2

    - uses: actions/checkout@v3

      with:

        path: smithy-rs

    - uses: actions/cache@v2

      name: Gradle Cache

      with:

        path: |

          gradle/caches

          gradle/wrapper

        key: ${{ runner.os }}-gradle-${{ hashFiles('gradle/caches/**/*', 'gradle/wrapper/**/*') }}

        restore-keys: |

          ${{ runner.os }}-gradle-

    # Pinned to the commit hash of v1.3.0

    - uses: Swatinem/rust-cache@842ef286fff290e445b90b4002cc9807c3669641

      with:

        sharedKey: ${{ runner.os }}-${{ env.rust_version }}-${{ github.job }}

        target-dir: ./smithy-rs-target

    - name: Download artifacts-generate-aws-sdk

      uses: actions/download-artifact@v3

      with:

        name: artifacts-generate-aws-sdk

    - name: Download artifacts-generate-aws-sdk-smoketest

      uses: actions/download-artifact@v3

      with:

        name: artifacts-generate-aws-sdk-smoketest

    - name: Untar artifacts

      run: |

        tar xfz artifacts-generate-aws-sdk.tar.gz

        tar xfz artifacts-generate-aws-sdk-smoketest.tar.gz

    - name: Download base image

      if: ${{ needs.acquire-base-image.outputs.image-in-artifacts == 'true' }}

      uses: actions/download-artifact@v3

      with:

        name: smithy-rs-base-image

    - name: Prepare build image

      run: |

        if [[ "${{ needs.acquire-base-image.outputs.image-in-artifacts }}" == "true" ]]; then

          docker load -i smithy-rs-base-image

        else

          ./smithy-rs/tools/ci-build/acquire-base-image --force-remote

        fi

        ./smithy-rs/tools/ci-build/create-local-build-image

    # This runs the commands from the matrix strategy

    - name: Run ${{ matrix.test.action }}

      run: ./smithy-rs/tools/ci-build/ci-action ${{ matrix.test.action }}

      uses: ./smithy-rs/.github/actions/docker-build

      with:

        action: ${{ matrix.test.action }}



  test-rust-windows:

    name: Rust Tests on Windows
@@ -187,7 +107,7 @@ jobs: 
      RUSTDOCFLAGS: -D warnings

      RUSTFLAGS: -D warnings

    steps:

    - uses: actions/checkout@v2

    - uses: actions/checkout@v3

      # Pinned to the commit hash of v1.3.0

    - uses: Swatinem/rust-cache@842ef286fff290e445b90b4002cc9807c3669641

      with:
@@ -208,6 +128,7 @@ jobs: 
          popd &>/dev/null

        done



  # This job is split out from the rest since it is not required to pass for merge

  check-sdk-examples:

    name: Check SDK Examples

    needs:
@@ -215,44 +136,13 @@ jobs: 
    - generate

    runs-on: ubuntu-latest

    steps:

    - uses: actions/checkout@v2

    - uses: actions/checkout@v3

      with:

        path: smithy-rs

    - uses: actions/cache@v2

      name: Gradle Cache

      with:

        path: |

          gradle/caches

          gradle/wrapper

        key: ${{ runner.os }}-gradle-${{ hashFiles('gradle/caches/**/*', 'gradle/wrapper/**/*') }}

        restore-keys: |

          ${{ runner.os }}-gradle-

      # Pinned to the commit hash of v1.3.0

    - uses: Swatinem/rust-cache@842ef286fff290e445b90b4002cc9807c3669641

      with:

        sharedKey: ${{ runner.os }}-${{ env.rust_version }}-${{ github.job }}

        target-dir: ./smithy-rs-target

    - name: Download artifacts-generate-aws-sdk

      uses: actions/download-artifact@v3

      with:

        name: artifacts-generate-aws-sdk

    - name: Untar artifacts

      run: tar xfz artifacts-generate-aws-sdk.tar.gz

    - name: Download build image

      if: ${{ needs.acquire-base-image.outputs.image-in-artifacts == 'true' }}

      uses: actions/download-artifact@v3

    - name: Run ${{ matrix.actions.action }}

      uses: ./smithy-rs/.github/actions/docker-build

      with:

        name: smithy-rs-base-image

    - name: Prepare build image

      run: |

        if [[ "${{ needs.acquire-base-image.outputs.image-in-artifacts }}" == "true" ]]; then

          docker load -i smithy-rs-base-image

        else

          ./smithy-rs/tools/ci-build/acquire-base-image --force-remote

        fi

        ./smithy-rs/tools/ci-build/create-local-build-image

    - name: Run check-aws-sdk-examples

      run: ./smithy-rs/tools/ci-build/ci-action check-aws-sdk-examples

        action: check-aws-sdk-examples



  # Pseudo-job that depends on matrix jobs so that we don't have to enter

  # the myriad of test matrix combinations into GitHub's protected branch rules

tools/ci-build/README.md

+4 −5
Original line number Diff line number Diff line
@@ -4,9 +4,9 @@ ci-build 
This directory includes the build Docker image and scripts to use it in CI.

- `../Dockerfile`: Dockerfile used to create the base build image. Needs to be in `tools/` root so that it

  can copy all the tools source code into the image at build time.

- `acquire-base-image`: Script that retrieves the base build image from public ECR or creates one locally

  depending on the state of the tools directory. If the tools have changed (in git history), then it opts

  to create a new image rather than reuse an existing one.

- `acquire-build-image`: Script that retrieves the base build image from public ECR or creates one locally

  depending on the state of the tools directory. If the git hash of the tools directory doesn't exist as a tag

  in public ECR, then it will build a new image locally. Otherwise, it will download the existing one and reuse it.

- `add-local-user.dockerfile`: Creates a user in the build image with the host's user ID

- `build.docker-compose.yml`: Docker Compose file for using the build image

- `ci-action`: Script for running CI actions inside of the Docker build image
@@ -36,5 +36,4 @@ $ORIGIN_PATH/tools/ci-build/ci-action <action> [args...] 
The action names are the names of the scripts in `scripts/`, and `[args...]` get forwarded to those scripts.



__Note:__ `ci-action` does not rebuild the build image, so if you modified a script,

you need to run `./acquire-base-image --force-local && ./create-local-build-image` from

the origin `tools/ci-build` path.
 
you need to run `./acquire-build-image` from the origin `tools/ci-build` path.