Unverified Commit 36a82ddd authored by Aaron Todd's avatar Aaron Todd Committed by GitHub
Browse files

refactor to use private ecr repository (#4153) 

## Description
Refactor to use private ECR repository. The primary motivation for this
is to optimize our internal release process to use the same build image.
Ideally we'd use a public ECR repository still but they don't support
lifecycle rules for cleaning up old/unused build images (of which we
create a lot of).

- **ECR Repository Migration**: Switched from
public.ecr.aws/w0m4q9l7/github-awslabs-smithy-rs-ci to a private ECR
repository
- **IAM Role Update**: Updated from `SMITHY_RS_PUBLIC_ECR_PUSH_ROLE_ARN`
to `SMITHY_RS_ECR_PUSH_ROLE_ARN` secret
- **Image Tagging**: Changed image tag prefix from just `<tools-hash>`
hash to `ci-<tools-hash>` (prefix is used for ECR image lifecycle rules
and cleanup of old images)
- **New Upload Script**: Added `upload-build-image.sh` for uploading to
ECR repo
- **Enhanced acquire-build-image**: Improved the build image acquisition
script with better error handling and fallback mechanisms when not
authenticated
- **Documentation**: Added comprehensive README for GitHub Actions
scripts explaining usage, environment variables, and behavior

## Testing
* Tested from [fork](https://github.com/smithy-lang/smithy-rs/pull/4157)
* Tested [dry run
release](https://github.com/smithy-lang/smithy-rs/actions/runs/15586780061)


## Checklist
<!--- If a checkbox below is not applicable, then please DELETE it
rather than leaving it unchecked -->
- [ ] For changes to the smithy-rs codegen or runtime crates, I have
created a changelog entry Markdown file in the `.changelog` directory,
specifying "client," "server," or both in the `applies_to` key.
- [ ] For changes to the AWS SDK, generated SDK code, or SDK runtime
crates, I have created a changelog entry Markdown file in the
`.changelog` directory, specifying "aws-sdk-rust" in the `applies_to`
key.

----

_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
parent 54d26cbb

.github/actions/docker-build/action.yml

+6 −1
Original line number Diff line number Diff line
@@ -43,6 +43,8 @@ runs: 
    shell: bash

    run: |

      set -x

      ls -lsa

      docker image ls



      # Check the build artifacts to see if a prior step built a new Docker build image.

      # If smithy-rs-base-image was included in the downloaded build artifacts, then load
@@ -50,11 +52,14 @@ runs: 
      # from attempting to download an image from ECR since it will already exist,

      # which enables testing build image modifications as part of the pull request.

      if [[ -d smithy-rs-base-image ]]; then

        IMAGE_TAG="$(./smithy-rs/.github/scripts/docker-image-hash)"

        echo "found base image in artifacts...loading it back into docker"

        IMAGE_TAG="ci-$(./smithy-rs/.github/scripts/docker-image-hash)"

        docker load -i smithy-rs-base-image/smithy-rs-base-image

        docker tag "smithy-rs-base-image:${IMAGE_TAG}" "smithy-rs-base-image:local"

      fi



      docker image ls



      # For this step, we want images to come from build artifacts (built as part a prior step),

      # or from ECR. We disable building the image from scratch so that any mistakes in the CI

      # configuration won't cause each individual action to build its own image, which would

.github/scripts/README.md

0 → 100644
+128 −0
Original line number Diff line number Diff line 
# GitHub Actions Scripts



This directory contains scripts used in the smithy-rs CI/CD workflows.



## acquire-build-image



**Purpose**: Acquires and prepares Docker build images for CI/CD workflows. Acts as an intelligent wrapper that handles

image availability checking, remote pulling from AWS ECR, authentication, and local building as needed.



### Usage



```bash

# Basic usage

./acquire-build-image



# Run self-tests

./acquire-build-image --self-test



# With environment variables

ALLOW_LOCAL_BUILD=false ./acquire-build-image

GITHUB_ACTIONS=true ./acquire-build-image

```



### Environment Variables



- `ALLOW_LOCAL_BUILD` - Enable local image building (default: `true`)

- `GITHUB_ACTIONS` - Indicates running in GitHub Actions (default: `false`)

- `ENCRYPTED_DOCKER_PASSWORD` - Base64 encrypted Docker registry password

- `DOCKER_LOGIN_TOKEN_PASSPHRASE` - Passphrase for decrypting Docker password

- `OCI_EXE` - Docker-compatible executable e.g. docker, finch, podman, etc (default: `docker`)



### Behavior & Outputs



#### When image exists locally:

- Uses existing local image

- Tags: `smithy-rs-base-image:local`, `smithy-rs-build-image:latest`

- No network activity



#### When image doesn't exist locally:

1. **Attempts remote pull** from AWS ECR (`<acccount-id>.dkr.ecr.us-west-2.amazonaws.com/smithy-rs-build-image`)

   - On success: Tags remote image as `smithy-rs-base-image:<tag>`

   - On failure: Falls back to local build (if enabled)



2. **Local build fallback** (when remote pull fails or on ARM64):

   - Builds from `tools/ci-build/Dockerfile`

   - Tags: `smithy-rs-base-image:<tag>`

   - In GitHub Actions: Also saves image to `./smithy-rs-base-image` file



3. **Final step** (always):

   - Creates user-specific build image: `smithy-rs-build-image:latest`

   - Tags base image as: `smithy-rs-base-image:local`



### Exit Codes



- `0` - Success: Build image ready for use

- `1` - Failure: Unable to acquire image



### Common Scenarios



**Local development (first run):**

```bash

./acquire-build-image

# → Pulls remote image → Tags locally → Creates build image

```



**Local development (subsequent runs):**

```bash

./acquire-build-image

# → Uses local image → Creates build image (fast)

```



**GitHub Actions:**

```bash

GITHUB_ACTIONS=true ./acquire-build-image

# → Same as above, but saves image file for job sharing if built locally

```



**ARM64 systems (Apple Silicon):**

```bash

./acquire-build-image

# → Skips remote pull → Builds locally (architecture mismatch)

```



## docker-image-hash



**Purpose**: Generates a unique hash based on the contents of the `tools/ci-build` directory. Used to create consistent Docker image tags.



### Usage



```bash

./docker-image-hash

# Outputs: a1b2c3d4e5f6... (git hash of tools/ci-build directory)

```



### Output

- Prints a git hash to stdout based on all files in `tools/ci-build`

- Hash changes only when build configuration files are modified

- Used by `acquire-build-image` to determine image tags



## upload-build-image.sh



**Purpose**: Uploads a local Docker build image to AWS ECR with proper authentication and tagging.



### Usage



```bash

# Upload with specific tag

./upload-build-image.sh <tag-name>



# Dry run (skip actual push)

DRY_RUN=true ./upload-build-image.sh <tag-name>



# Use alternative OCI executable

OCI_EXE=podman ./upload-build-image.sh <tag-name>

```



### Environment Variables

- `DRY_RUN` - Skip push to ECR (default: `false`)

- `OCI_EXE` - Docker-compatible executable (default: `docker`)



### Behavior

1. Authenticates with AWS ECR using AWS CLI

2. Tags local `smithy-rs-build-image:latest` as `<account>.dkr.ecr.us-west-2.amazonaws.com/smithy-rs-build-image:<tag>`

3. Pushes tagged image to ECR (unless `DRY_RUN=true`)



### Requirements

- AWS CLI configured with appropriate permissions

- Local image `smithy-rs-build-image:latest` must exist

.github/scripts/acquire-build-image

+30 −16
Original line number Diff line number Diff line
@@ -13,9 +13,11 @@ import time 
import unittest

import base64



REMOTE_BASE_IMAGE_NAME = "public.ecr.aws/w0m4q9l7/github-awslabs-smithy-rs-ci"

AWS_ACCOUNT_ID="686190543447"

AWS_REGION="us-west-2"

ECR_REPOSITORY="smithy-rs-build-image"

REMOTE_BASE_IMAGE_NAME = f"{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.amazonaws.com/{ECR_REPOSITORY}"

LOCAL_BASE_IMAGE_NAME = "smithy-rs-base-image"

BUILD_IMAGE_NAME = "smithy-rs-build-image"

LOCAL_TAG = "local"



C_YELLOW = '\033[1;33m'
@@ -33,6 +35,7 @@ class DockerPullResult(Enum): 
    RETRYABLE_ERROR = 4

    NOT_FOUND = 5

    UNKNOWN_ERROR = 6

    UNAUTHENTICATED = 7





class Platform(Enum):
@@ -40,6 +43,9 @@ class Platform(Enum): 
    ARM_64 = 1





def oci_exe():

    return os.getenv("OCI_EXE", "docker")



# Script context

class Context:

    def __init__(self, start_path, script_path, tools_path, user_id, image_tag, allow_local_build, github_actions,
@@ -61,7 +67,8 @@ class Context: 
        script_path = os.path.dirname(os.path.realpath(__file__))

        tools_path = get_cmd_output("git rev-parse --show-toplevel", cwd=script_path)[1] + "/tools"

        user_id = get_cmd_output("id -u")[1]

        image_tag = get_cmd_output("./docker-image-hash", cwd=script_path)[1]

        tools_image_hash = get_cmd_output("./docker-image-hash", cwd=script_path)[1]

        image_tag = f"ci-{tools_image_hash}"

        allow_local_build = os.getenv("ALLOW_LOCAL_BUILD") != "false"

        github_actions = os.getenv("GITHUB_ACTIONS") == "true"

        encrypted_docker_password = os.getenv("ENCRYPTED_DOCKER_PASSWORD") or None
@@ -96,53 +103,58 @@ class Shell: 


    # Returns True if the given `image_name` and `image_tag` exist locally

    def docker_image_exists_locally(self, image_name, image_tag):

        (status, _, _) = get_cmd_output(f"docker inspect \"{image_name}:{image_tag}\"", check=False)

        (status, _, _) = get_cmd_output(f"{oci_exe()} inspect \"{image_name}:{image_tag}\"", check=False)

        return status == 0



    def docker_login(self, password):

        get_cmd_output("docker login --username AWS --password-stdin public.ecr.aws", input=password.encode('utf-8'))

        get_cmd_output(f"{oci_exe()} login --username AWS --password-stdin {AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.amazonaws.com", input=password.encode('utf-8'))



    # Pulls the requested `image_name` with `image_tag`. Returns `DockerPullResult`.

    def docker_pull(self, image_name, image_tag):

        (status, stdout, stderr) = get_cmd_output(f"docker pull \"{image_name}:{image_tag}\"", check=False)

        print("Docker pull output:")

        print("-------------------")

        (status, stdout, stderr) = get_cmd_output(f"{oci_exe()} pull \"{image_name}:{image_tag}\"", check=False)

        print(f"{oci_exe()} pull output:")

        print("------STDOUT---------")

        print(stdout)

        print("-------------------")

        print("------STDERR---------")

        print(stderr)

        print("-------------------")



        not_found_messages = ["not found: manifest unknown"]

        throttle_messages = ["toomanyrequests:"]

        retryable_messages = ["net/http: TLS handshake timeout"]

        unauthenticated_messages = ["no basic auth credentials"]

        if status == 0:

            return DockerPullResult.SUCCESS

        elif output_contains_any(stdout, stderr, throttle_messages):



        print(f"{oci_exe()} pull status: {status}")

        if output_contains_any(stdout, stderr, throttle_messages):

            return DockerPullResult.ERROR_THROTTLED

        elif output_contains_any(stdout, stderr, not_found_messages):

            return DockerPullResult.NOT_FOUND

        elif output_contains_any(stdout, stderr, retryable_messages):

            return DockerPullResult.RETRYABLE_ERROR

        elif output_contains_any(stdout, stderr, unauthenticated_messages):

            return DockerPullResult.UNAUTHENTICATED

        return DockerPullResult.UNKNOWN_ERROR



    # Builds the base image with the Dockerfile in `path` and tags with with `image_tag`

    def docker_build_base_image(self, image_tag, path):

        run(f"docker build -t \"smithy-rs-base-image:{image_tag}\" .", cwd=path)

        run(f"{oci_exe()} build -t \"smithy-rs-base-image:{image_tag}\" .", cwd=path)



    # Builds the local build image

    def docker_build_build_image(self, user_id, docker_image_path):

        run(

            f"docker build -t smithy-rs-build-image --file add-local-user.dockerfile --build-arg=USER_ID={user_id} .",

            f"{oci_exe()} build -t smithy-rs-build-image --file add-local-user.dockerfile --build-arg=USER_ID={user_id} .",

            cwd=docker_image_path

        )



    # Saves the Docker image named `image_name` with `image_tag` to `output_path`

    def docker_save(self, image_name, image_tag, output_path):

        run(f"docker save -o \"{output_path}\" \"{image_name}:{image_tag}\"")

        run(f"{oci_exe()} save -o \"{output_path}\" \"{image_name}:{image_tag}\"")



    # Tags an image with a new image name and tag

    def docker_tag(self, image_name, image_tag, new_image_name, new_image_tag):

        run(f"docker tag \"{image_name}:{image_tag}\" \"{new_image_name}:{new_image_tag}\"")

        run(f"{oci_exe()} tag \"{image_name}:{image_tag}\" \"{new_image_name}:{new_image_tag}\"")





# Pulls a Docker image and retries if it gets throttled
@@ -153,7 +165,7 @@ def docker_pull_with_retry(shell, image_name, image_tag, throttle_sleep_time=120 
        announce(f"Attempting to pull remote image {image_name}:{image_tag} (attempt {attempt})...")

        result = shell.docker_pull(image_name, image_tag)

        if result == DockerPullResult.ERROR_THROTTLED:

            announce("Docker pull failed due to throttling. Waiting and trying again...")

            announce("Pull failed due to throttling. Waiting and trying again...")

            time.sleep(throttle_sleep_time)

        elif result == DockerPullResult.RETRYABLE_ERROR:

            announce("A retryable error occurred. Trying again...")
@@ -198,7 +210,7 @@ def decrypt_and_login(shell, secret, passphrase): 
        ["gpg", "--decrypt", "--batch", "--quiet", "--passphrase", passphrase, "--output", "-"],

        input=decoded)

    shell.docker_login(password)

    print("Docker login success!")

    print(f"{oci_exe()} login success!")





def acquire_build_image(context=Context.default(), shell=Shell()):
@@ -214,6 +226,8 @@ def acquire_build_image(context=Context.default(), shell=Shell()): 
            elif pull_result == DockerPullResult.UNKNOWN_ERROR:

                announce("An unknown failure happened during Docker pull. This needs to be examined.")

                return 1

            elif pull_result == DockerPullResult.UNAUTHENTICATED:

                announce("Unable to authenticate and pull image from remote repository. A local build is required.")

            else:

                announce("Failed to pull remote image, which can happen if it doesn't exist.")

.github/scripts/upload-build-image.sh

0 → 100755
+52 −0
Original line number Diff line number Diff line 
#!/bin/bash

#

# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

# SPDX-License-Identifier: Apache-2.0

set -eux



if [ $# -ne 1 ]; then

    echo "Error: Tag name is required"

    echo "Usage: $0 <tag-name>"

    exit 1

fi



# Set OCI executor - default to docker if not set

: "${OCI_EXE:=docker}"



DRY_RUN=${DRY_RUN:-false}

TAG_NAME=$1

AWS_REGION="us-west-2"

AWS_ACCOUNT_ID="686190543447"

ECR_REPOSITORY="smithy-rs-build-image"

ECR_IMAGE="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${ECR_REPOSITORY}:${TAG_NAME}"



echo "Logging in to Amazon ECR..."

aws ecr get-login-password --region ${AWS_REGION} | ${OCI_EXE} login --username AWS --password-stdin ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com



if [ $? -ne 0 ]; then

    echo "Error: Failed to login to ECR"

    exit 1

fi



echo "Tagging image as: ${ECR_IMAGE}"

${OCI_EXE} tag ${ECR_REPOSITORY}:latest ${ECR_IMAGE}



if [ $? -ne 0 ]; then

    echo "Error: Failed to tag the image"

    exit 1

fi



if [[ "${DRY_RUN}" == "true" ]]; then

  echo "Dry run enabled - skipping push to ECR"

  exit 0

else

  echo "Pushing image to ECR..."

  ${OCI_EXE} push ${ECR_IMAGE}



  if [ $? -ne 0 ]; then

      echo "Error: Failed to push the image"

      exit 1

  fi



  echo "Successfully uploaded image to ECR: ${ECR_IMAGE}"

fi

.github/workflows/ci-main.yml

+4 −10
Original line number Diff line number Diff line
@@ -15,9 +15,6 @@ concurrency: 
  group: ci-main-yml

  cancel-in-progress: true



env:

  ecr_repository: public.ecr.aws/w0m4q9l7/github-awslabs-smithy-rs-ci



permissions:

  actions: read

  contents: read
@@ -42,14 +39,14 @@ jobs: 
    - name: Acquire credentials

      uses: aws-actions/configure-aws-credentials@v4

      with:

        role-to-assume: ${{ secrets.SMITHY_RS_PUBLIC_ECR_PUSH_ROLE_ARN }}

        role-to-assume: ${{ secrets.SMITHY_RS_ECR_PUSH_ROLE_ARN }}

        role-session-name: GitHubActions

        aws-region: us-west-2

    - name: Save the docker login password to the output

      id: set-token

      run: |

        ENCRYPTED_PAYLOAD=$(

          gpg --symmetric --batch --passphrase "${{ secrets.DOCKER_LOGIN_TOKEN_PASSPHRASE }}" --output - <(aws ecr-public get-login-password --region us-east-1) | base64 -w0

          gpg --symmetric --batch --passphrase "${{ secrets.DOCKER_LOGIN_TOKEN_PASSPHRASE }}" --output - <(aws ecr get-login-password --region us-west-2) | base64 -w0

        )

        echo "docker-login-password=$ENCRYPTED_PAYLOAD" >> $GITHUB_OUTPUT

    - name: Acquire base image
@@ -61,11 +58,8 @@ jobs: 
      run: ./.github/scripts/acquire-build-image

    - name: Tag and upload image

      run: |

        IMAGE_TAG="$(./.github/scripts/docker-image-hash)"

        docker tag "smithy-rs-base-image:${IMAGE_TAG}" "${{ env.ecr_repository }}:${IMAGE_TAG}"

        docker tag "smithy-rs-base-image:${IMAGE_TAG}" "${{ env.ecr_repository }}:main"

        docker push "${{ env.ecr_repository }}:${IMAGE_TAG}"

        docker push "${{ env.ecr_repository }}:main"

        IMAGE_TAG="ci-$(./.github/scripts/docker-image-hash)"

        ./smithy-rs/.github/scripts/upload-build-image.sh $IMAGE_TAG



  # Run the shared CI after a Docker build image has been uploaded to ECR

  ci: