RFC: Split the release process between `smithy-rs` and `aws-sdk-rust` (#970)

RFC: Split Release Process

==========================

> Status: RFC

At the time of writing, the `aws-sdk-rust` repository is used exclusively

for the entire release process of both the Rust runtime crates from `smithy-rs` as

well as the AWS runtime crates and the AWS SDK. This worked well when `smithy-rs` was

only used for the AWS SDK, but now that it's also used for server codegen, there

are issues around publishing the server-specific runtime crates since they don't

belong to the SDK.

This RFC proposes a new split-release process so that the entire `smithy-rs` runtime

can be published separately before the AWS SDK is published.

Terminology

-----------

- **Smithy Runtime Crate**: A crate that gets published to crates.io and supports

  the code generated by `smithy-rs`. These crates don't provide any SDK-only functionality.

  These crates can support client and/or server code, and clients or servers may use

  only a subset of them.

- **AWS Runtime Crate**: A crate of SDK-specific code that supports the code generated

  by the `aws/codegen` module in `smithy-rs`. These also get published to crates.io.

- **Publish-ready Bundle**: A build artifact that is ready to publish to crates.io without

  additional steps (such as running the publisher tool's `fix-manifests` subcommand). Publishing

  one group of crates before another is not considered an additional step for this definition.

- **Releaser**: A developer, automated process, or combination of the two that performs the actual release.

Requirements

------------

At a high level, the requirements are: publish from both `smithy-rs` and `aws-sdk-rust`

while preserving our current level of confidence in the quality of the release. This

can be enumerated as:

1. All Smithy runtime crates must be published together from `smithy-rs`

2. AWS runtime crates and the SDK must be published together from `aws-sdk-rust`

3. CI on `smithy-rs` must give confidence that the Smithy runtime crates,

   AWS runtime crates, and SDK are all at the right quality bar for publish.

4. CI on the `aws-sdk-rust` repository must give confidence that the AWS SDK and its

   runtime crates are at the right quality bar for publish. To do this successfully,

   it must run against the exact versions of the Smithy runtime crates the code was

   generated against _both before AND after they have been published to crates.io_.

Background: How Publishing Worked Before

----------------------------------------

The publish process to crates.io relied on copying all the Smithy runtime crates

into the final `aws-sdk-rust` repository. Overall, the process looked as follows:

1. `smithy-rs` generates a complete `aws-sdk-rust` source bundle at CI time

2. The releaser copies the generated bundle over to `aws-sdk-rust`

3. The releaser runs the `publisher fix-manifests` subcommand to correct the

   `Cargo.toml` files generated by `smithy-rs`

4. The `aws-sdk-rust` CI performs one last pass on the code to verify it's sound

5. The releaser runs the `publisher publish` subcommand to push all the crates up to crates.io

Proposed Solution

-----------------

CI in `smithy-rs` will be revised to generate two separate build artifacts where it generates

just an SDK artifact previously. Now, it will have two build targets that get executed from CI

to generate these artifacts:

- `rust-runtime:assemble` - Generates a publish-ready bundle of Smithy runtime crates.

- `aws:sdk:assemble` - Generates a publish-ready bundle of AWS runtime crates, SDK crates,

  and just the Smithy runtime crates that are used by the SDK.

The `aws-sdk-rust` repository will have a new `next` branch that has its own set of CI workflows

and branch protection rules. The releaser will take the `aws:sdk:assemble` artifact and apply it

directly to this `next` branch as would have previously been done against the `main` branch.

The `main` branch will continue to have the same CI as `next`.

When it's time to cut a release, the releaser will do the following:

1. Tag `smithy-rs` with the desired version number

2. Wait for CI to build artifacts for the tagged release

3. Pull-request the SDK artifacts over to `aws-sdk-rust/next` (this will be automated in the future)

6. Pull-request merge `aws-sdk-rust/next` into `aws-sdk-rust/main`

7. Wait for successful CI in `main`

8. Tag release for `main`

9. Publish SDK with publisher tool

The server team can then download the `rust-runtime:assemble` build artifact for the tagged release

in `smithy-rs`, and publish the `aws-smithy-http-server` crate from there.

### Avoiding mistakes by disallowing creation of publish-ready bundles outside of CI

It should be difficult to accidentally publish a locally built set of crates. To add friction to this,

the `smithy-rs` build process will look for the existence of the `GITHUB_ACTIONS=true` environment variable.

If this environment variable is not set, then it will pass a flag to the Rust codegen plugin that tells it to

emit a `publish = false` under `[package]` in the generated `Cargo.toml`.

This could be easily circumvented, but the goal is to reduce the chances of accidentally publishing

crates rather than making it impossible.

Alternatives Considered

-----------------------

### Publish Smithy runtime crates from `smithy-rs` build artifacts

This approach is similar to the proposed solution, except that the SDK would not publish

the Smithy runtime crates. The `aws-sdk-rust/main` branch would have a small tweak to its CI

so that the SDK is tested against the Smithy runtime crates that are published to crates.io

This CI process would look as follows:

1. Shallow clone `aws-sdk-rust` with the revision being tested

2. Run a script to remove the `path` argument for the Smithy runtime crate dependencies for every crate

   in `aws-sdk-rust`. For example,

```toml

aws-smithy-types = { version = "0.33.0", path = "../aws-smithy-types" }

```

Would become:

```toml

aws-smithy-types = { version = "0.33.0" }

```

3. Run the tests as usual

When it's time to cut a release, the releaser will do the following:

1. Tag `smithy-rs` with the desired version number

2. Wait for CI to build artifacts for the tagged release

3. Pull-request the SDK artifacts over to `aws-sdk-rust/next`

4. Wait for successful CI in `aws-sdk-rust/next`

5. Download the Smithy runtime crates build artifact and publish it to crates.io

6. Pull-request merge `aws-sdk-rust/next` into `aws-sdk-rust/main`

7. Wait for successful CI in `main` (this time actually running against the crates.io Smithy runtime crates)

8. Tag release for `main`

9. Publish SDK with publisher tool

### Keep Smithy runtime crates in `smithy-rs`

This approach is similar to the previous alternative, except that the `aws-sdk-rust` repository

won't have a snapshot of the Smithy runtime crates, and an additional step needs to be performed

during CI for the `next` branch so that it looks as follows:

1. Make a shallow clone of `aws-sdk-rust/next`

2. Retrieve the `smithy-rs` commit hash that was used to generate the SDK from a file

   that was generated alongside the rest of the build artifacts from `smithy-rs` and

   copied into `aws-sdk-rust`.

3. Make a shallow clone of `smithy-rs` at the correct commit hash

4. Use a script to add a `[patch]` section to all the AWS SDK crates to point to the

   Smithy runtime crates from the local clone of `smithy-rs`.

   For example:

```toml

# The dependencies section is left alone, but is here for context

[dependencies]

# Some version of aws-smithy-types that isn't on crates.io yet, referred to as `<unreleased>` below

aws-smithy-types = "<unreleased>"

# This patch section gets added by the script

[patch.crates-io]

aws-smithy-types = { version = "<unreleased>", path = "path/to/local/smithy-rs/rust-runtime/aws-smithy-types"}

```

5. Run CI as normal.

**Note:** `smithy-rs` would need to do the same patching in CI as `aws-sdk-rust/next` since the generated

SDK would not have path dependencies for the Smithy runtime crates (since they are a publish-ready bundle

intended for landing in `aws-sdk-rust`). The script that does this patching could live in `smithy-rs` and be

reused by `aws-sdk-rust`.

The disadvantage of this approach is that a customer having an issue with the current release wouldn't be able

to get a fix sooner by patching their own project's crate manifest to use the `aws-sdk-rust/next` branch before

a release is cut since their project wouldn't be able to find the unreleased Smithy runtime crates.

Changes Checklist

-----------------

- In `smithy-rs`:

  - [ ] Move publisher tool from `aws-sdk-rust` into `smithy-rs`

  - [ ] Modify `aws:sdk:assemble` target to run the publisher `fix-manifests` subcommand

  - [ ] Add `rust-runtime:assemble` target that generates publish-ready Smithy runtime crates

  - [ ] Add CI step to create Smithy runtime bundle artifact

  - [ ] Add `GITHUB_ACTIONS=true` env var check for setting the `publish` flag in generated AND runtime manifests

  - [ ] Revise publisher tool to publish from an arbitrary directory

- In `aws-sdk-rust`:

  - [ ] Implement CI for the `aws-sdk-rust/next` branch

  - [ ] Remove the publisher tool

- [ ] Update release process documentation