Unverified Commit 2016a67a authored by Aaron Todd's avatar Aaron Todd Committed by GitHub
Browse files

publish codegen artifacts to maven central (#4218) 

## Motivation and Context
Publish codegen artifacts to Maven Central so users can generate Rust
clients + servers without cloning smithy-rs and patching our build
files.

## Description
* Move dependency management to version catalogs 
* Move common kotlin and publishing configuration to conventional gradle
plugins removing duplication and making a single place to modify this
config
* Add jreleaser and configure it for releasing to maven central (this
seems to be where everyone is landing, smithy team uses it already and
Kotlin SDK is migrating)
* Introduce tasks for checking if a codegen version exists and if
`gradle.properties` has been updated if any codegen project has changed
    * On release it checks if we need to do a publish
* On PR it checks if we modified codegen projects that we have also
bumped the version
* Rename several modules
    * `codegen-server/python` -> `codegen-server/codegen-server-python`
* `codegen-server/typescript` ->
`codegen-server/codegen-server-typescript`
    *  `aws/sdk-codegen` -> `aws/codegen-aws-sdk`

## TODO
* ~Need to decide on artifact names. We can probably live with most of
them but the python/typescript artifacts don't indicate anything server
specific and the `sdk-codegen` one doesn't indicate "AWS SDK" specific.
Generally we don't publish the AWS SDK specific codegen but we tied up a
lot of the "AWS specific" stuff like sigv in that module so seems like
we might need to publish it as well for it to be of use. Open to
discussion though.~
* Need to work with @drganjoo on internal testing, in particular need to
figure out internal patch file updates needed to land this and
versioning scheme
* ~Need to finish configuring secrets~
* ~Publish GPG keys, for some reason this is still not working I may
need to switch machines to try it on~
* ~Complete additional dry run release testing~
* Test artifacts and document how to use them in a build script to
generate code for generic smithy models

## Testing
* Basic testing of gradle tasks done against already published artifacts
for other codegenerators, seems to be working but may still need refined
* Test dry run release
*
https://github.com/smithy-lang/smithy-rs/actions/runs/16445373815/job/46479485851
## Checklist
<!--- If a checkbox below is not applicable, then please DELETE it
rather than leaving it unchecked -->
- [x] For changes to the smithy-rs codegen or runtime crates, I have
created a changelog entry Markdown file in the `.changelog` directory,
specifying "client," "server," or both in the `applies_to` key.

----

_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
parent 13f00bdc

.github/workflows/ci.yml

+2 −0
Original line number Diff line number Diff line
@@ -127,6 +127,8 @@ jobs: 
          runner: smithy_ubuntu-latest_8-core

        - action: check-deterministic-codegen

          runner: smithy_ubuntu-latest_8-core

        - action: check-codegen-version

          runner: ubuntu-latest

    steps:

    - uses: GitHubSecurityLab/actions-permissions/monitor@v1

    - uses: actions/checkout@v4

.github/workflows/dry-run-release.yml

+4 −0
Original line number Diff line number Diff line
@@ -43,3 +43,7 @@ jobs: 
      CANARY_GITHUB_ACTIONS_ROLE_ARN: ${{ secrets.CANARY_GITHUB_ACTIONS_ROLE_ARN }}

      CANARY_STACK_CDK_OUTPUTS_BUCKET_NAME: ${{ secrets.CANARY_STACK_CDK_OUTPUTS_BUCKET_NAME }}

      SMITHY_RS_ECR_PUSH_ROLE_ARN: ${{ secrets.SMITHY_RS_ECR_PUSH_ROLE_ARN }}

      MAVEN_CENTRAL_GPG_PUBLIC_KEY_SECRET_ARN: ${{ secrets.MAVEN_CENTRAL_GPG_PUBLIC_KEY_SECRET_ARN }}

      MAVEN_CENTRAL_GPG_PRIVATE_KEY_SECRET_ARN: ${{ secrets.MAVEN_CENTRAL_GPG_PRIVATE_KEY_SECRET_ARN }}

      MAVEN_CENTRAL_GPG_PASSPHRASE_SECRET_ARN: ${{ secrets.MAVEN_CENTRAL_GPG_PASSPHRASE_SECRET_ARN }}

      MAVEN_CENTRAL_SONATYPE_CREDENTIALS_SECRET_ARN: ${{ secrets.MAVEN_CENTRAL_SONATYPE_CREDENTIALS_SECRET_ARN }}

.github/workflows/prod-release.yml

+4 −0
Original line number Diff line number Diff line
@@ -37,3 +37,7 @@ jobs: 
      CANARY_GITHUB_ACTIONS_ROLE_ARN: ${{ secrets.CANARY_GITHUB_ACTIONS_ROLE_ARN }}

      CANARY_STACK_CDK_OUTPUTS_BUCKET_NAME: ${{ secrets.CANARY_STACK_CDK_OUTPUTS_BUCKET_NAME }}

      SMITHY_RS_ECR_PUSH_ROLE_ARN: ${{ secrets.SMITHY_RS_ECR_PUSH_ROLE_ARN }}

      MAVEN_CENTRAL_GPG_PUBLIC_KEY_SECRET_ARN: ${{ secrets.MAVEN_CENTRAL_GPG_PUBLIC_KEY_SECRET_ARN }}

      MAVEN_CENTRAL_GPG_PRIVATE_KEY_SECRET_ARN: ${{ secrets.MAVEN_CENTRAL_GPG_PRIVATE_KEY_SECRET_ARN }}

      MAVEN_CENTRAL_GPG_PASSPHRASE_SECRET_ARN: ${{ secrets.MAVEN_CENTRAL_GPG_PASSPHRASE_SECRET_ARN }}

      MAVEN_CENTRAL_SONATYPE_CREDENTIALS_SECRET_ARN: ${{ secrets.MAVEN_CENTRAL_SONATYPE_CREDENTIALS_SECRET_ARN }}

.github/workflows/release.yml

+83 −0
Original line number Diff line number Diff line
@@ -39,6 +39,14 @@ on: 
        required: true

      SMITHY_RS_ECR_PUSH_ROLE_ARN:

        required: true

      MAVEN_CENTRAL_GPG_PUBLIC_KEY_SECRET_ARN:

        required: true

      MAVEN_CENTRAL_GPG_PRIVATE_KEY_SECRET_ARN:

        required: true

      MAVEN_CENTRAL_GPG_PASSPHRASE_SECRET_ARN:

        required: true

      MAVEN_CENTRAL_SONATYPE_CREDENTIALS_SECRET_ARN:

        required: true



jobs:

  check-actor-for-prod-run:
@@ -273,6 +281,81 @@ jobs: 
            releaseCommitish: "${{ steps.push-changelog.outputs.commit_sha }}"

          });



  publish-to-maven-central:

    name: Publish Codegen artifacts to Maven Central

    needs:

    - release

    if: always() && needs.release.result == 'success'

    runs-on: ubuntu-latest

    steps:

    - uses: GitHubSecurityLab/actions-permissions/monitor@v1

    - uses: actions/checkout@v4

      with:

        path: smithy-rs

        ref: ${{ inputs.commit_sha }}

        fetch-depth: 0

    - name: Set up JDK

      uses: actions/setup-java@v4

      with:

        distribution: temurin

        java-version: '17'

    - name: Check if publishing is needed

      id: check-publish

      shell: bash

      working-directory: smithy-rs

      run: |

        # Run the Gradle task to check if publishing is needed

        ./gradlew checkMavenCentralPublishingNeeded



        # Read the result from the build

        if grep -q "mavenCentralPublishingNeeded=true" build/maven-central/publishing.properties; then

          echo "publish=true" >> $GITHUB_OUTPUT

        else

          echo "publish=false" >> $GITHUB_OUTPUT

        fi

    - name: Acquire credentials

      if: steps.check-publish.outputs.publish == 'true'

      uses: aws-actions/configure-aws-credentials@v4

      with:

        role-to-assume: ${{ secrets.SMITHY_RS_ECR_PUSH_ROLE_ARN }}

        role-session-name: GitHubActions

        aws-region: us-west-2

    - name: Publish to Maven Central

      if: steps.check-publish.outputs.publish == 'true'

      shell: bash

      working-directory: smithy-rs

      env:

        GPG_PUBLIC_KEY_SECRET_ARN: ${{ secrets.MAVEN_CENTRAL_GPG_PUBLIC_KEY_SECRET_ARN }}

        GPG_PRIVATE_KEY_SECRET_ARN: ${{ secrets.MAVEN_CENTRAL_GPG_PRIVATE_KEY_SECRET_ARN }}

        GPG_PASSPHRASE_SECRET_ARN: ${{ secrets.MAVEN_CENTRAL_GPG_PASSPHRASE_SECRET_ARN }}

        SONATYPE_CREDENTIALS_SECRET_ARN: ${{ secrets.MAVEN_CENTRAL_SONATYPE_CREDENTIALS_SECRET_ARN }}

        JRELEASER_DRY_RUN: ${{ inputs.dry_run }}

      run: |

        pwd

        # Get secrets from AWS Secrets Manager

        GPG_PUBLIC_KEY=$(aws secretsmanager get-secret-value --secret-id $GPG_PUBLIC_KEY_SECRET_ARN --query SecretString --output text)

        GPG_PRIVATE_KEY=$(aws secretsmanager get-secret-value --secret-id $GPG_PRIVATE_KEY_SECRET_ARN --query SecretString --output text)

        GPG_PASSPHRASE=$(aws secretsmanager get-secret-value --secret-id $GPG_PASSPHRASE_SECRET_ARN --query SecretString --output text)



        # Get Sonatype credentials from JSON secret

        SONATYPE_CREDS=$(aws secretsmanager get-secret-value --secret-id $SONATYPE_CREDENTIALS_SECRET_ARN --query SecretString --output text)

        MAVEN_CENTRAL_USERNAME=$(echo $SONATYPE_CREDS | jq -r '.["sonatype-portal-token-username"]')

        MAVEN_CENTRAL_TOKEN=$(echo $SONATYPE_CREDS | jq -r '.["sonatype-portal-token"]')



        # Set up JReleaser environment variables

        export JRELEASER_GPG_PUBLIC_KEY="$GPG_PUBLIC_KEY"

        export JRELEASER_GPG_SECRET_KEY="$GPG_PRIVATE_KEY"

        export JRELEASER_GPG_PASSPHRASE="$GPG_PASSPHRASE"

        export JRELEASER_MAVENCENTRAL_USERNAME="$MAVEN_CENTRAL_USERNAME"

        export JRELEASER_MAVENCENTRAL_TOKEN="$MAVEN_CENTRAL_TOKEN"

        export JRELEASER_GENERIC_TOKEN=not-used-but-must-be-set



        # Run Gradle publish task to stage outputs to build/m2 directory

        ./gradlew publish

        ls -lsa build/m2/software/amazon/smithy/rust

        ./gradlew jreleaserConfig

        ./gradlew jreleaserFullRelease



  # If this step fails for any reason, there's no need to retry the release workflow, as this step is auxiliary

  # and the release itself was successful. Instead, manually trigger `backport-pull-request.yml`.

  open-backport-pull-request:

README.md

+4 −4
Original line number Diff line number Diff line
@@ -34,10 +34,10 @@ Project Layout 
  installed for FIP support to compile properly)

  * `./gradlew :aws:sdk:{cargoCheck, cargoTest, cargoDocs, cargoClippy}`: Generate & run specified cargo command.

* `codegen-core`: Common code generation logic useful for clients and servers

* `codegen-client`: Whitelabel Smithy client code generation

* `codegen-client`: Smithy client code generation

* `codegen-client-test`: Smithy protocol test generation & integration tests for Smithy client whitelabel code

* [`design`](design): Design documentation. See the [design/README.md](design/README.md) for details about building / viewing.

* `codegen-server`: Whitelabel Smithy server code generation

* `codegen-server`: Smithy server code generation

* `codegen-server-test`: Smithy protocol test generation & integration tests for Smithy server whitelabel code

* `examples`: A collection of server implementation examples
@@ -53,7 +53,7 @@ In general, the components of smithy-rs affect each other in the following order 
1. `rust-runtime`

2. `codegen` and `codegen-server`

3. `aws/rust-runtime`

4. `aws/sdk-codegen`

4. `aws/codegen-aws-sdk`



Some components, such as `codegen-client-test` and `codegen-server-test`, are purely for testing other components.
@@ -120,7 +120,7 @@ to generate more or less AWS service clients. 


```bash

# Run Kotlin codegen unit tests

./gradlew aws:sdk-codegen:check

./gradlew aws:codegen-aws-sdk:check

# Generate an SDK, but do not attempt to compile / run tests. Useful for inspecting generated code

./gradlew :aws:sdk:assemble

# Run all the tests