Enhance gradle tasks for managing lockfiles (#3829)
## Description This PR introduces and updates gradle tasks for managing lockfiles. Here are the highlights: - [The SDK lockfile](https://github.com/smithy-lang/smithy-rs/blob/main/aws/sdk/Cargo.lock) can now be generated directly within the `smithy-rs` repository without the `aws-sdk-rust` repository. - The SDK lockfile can be synchronized with runtime lockfiles, updating only new dependencies while preserving the versions of existing ones. - To prevent updating broken dependencies to the latest versions, we track the last known good versions and downgrade them to those versions. New/updated gradle tasks are intended for automation: - This existing task no longer requires `-Paws-sdk-rust-path`. We plan to incorporate it into a weekly GitHub Action to automate lockfile updates: ``` ./gradlew aws:sdk:cargoUpdateAllLockfiles ``` - This new task synchronizes the SDK lockfile with runtime lockfiles. We plan to integrate it into pre-commit hooks: ``` ./gradlew aws:sdk:syncAwsSdkLockfile ``` In addition, this PR has updated the SDK lockfile by executing `./gradlew aws:sdk:syncAwsSdkLockfile`. The updated lockfile no longer includes many SDK crates that are unused in CI/CD processes. The new SDK lockfile is in sync with the runtime lockfiles: ``` ➜ smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) sdk-lockfiles audit 2024-09-12T16:02:25.193765Z INFO sdk_lockfiles::audit: checking whether `rust-runtime/Cargo.lock` is covered by the SDK lockfile... 2024-09-12T16:02:25.224862Z INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/Cargo.lock` is covered by the SDK lockfile... 2024-09-12T16:02:25.225389Z INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/aws-config/Cargo.lock` is covered by the SDK lockfile... SUCCESS ``` ## Testing I have verified the change against basic use cases: #### When running `cargoUpdateAllLockfiles`, dependencies will be updated to their latest versions, while broken crates will be pinned to the last known good versions. <details> <summary> Expand for more details...</summary> When we execute ``` smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ ./gradlew aws:sdk:cargoUpdateAllLockfiles ... BUILD SUCCESSFUL in 1m 7s ``` all lockfiles include the latest versions of dependencies, except for those that are pinned due to being broken. Currently, minicbor is [pinned to 0.24.2](https://github.com/smithy-lang/smithy-rs/blob/7f1d992214c25347fffeea63f8ceb58f77c7a8bb/aws/sdk/build.gradle.kts#L503-L504): ``` ➜ smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ git status On branch ysaito/enhance-gradle-tasks-for-lockfile Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: aws/rust-runtime/Cargo.lock modified: aws/rust-runtime/aws-config/Cargo.lock modified: aws/sdk/Cargo.lock modified: rust-runtime/Cargo.lock no changes added to commit (use "git add" and/or "git commit -a") ``` ``` ➜ smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ cat aws/sdk/Cargo.lock | rg -C1 minicbor ... --- [[package]] name = "minicbor" version = "0.24.2" --- ... ➜ smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ cat rust-runtime/Cargo.lock | rg -C1 minicbor ... --- [[package]] name = "minicbor" version = "0.24.2" --- ... ``` Finally, the `sdk-lockfiles audit` command should run successfully after updating all lockfiles: ``` ➜ smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ sdk-lockfiles audit 2024-09-12T15:35:47.890530Z INFO sdk_lockfiles::audit: checking whether `rust-runtime/Cargo.lock` is covered by the SDK lockfile... 2024-09-12T15:35:47.922468Z INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/Cargo.lock` is covered by the SDK lockfile... 2024-09-12T15:35:47.922898Z INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/aws-config/Cargo.lock` is covered by the SDK lockfile... SUCCESS ``` I also specified multiple broken dependencies and verified they were all downgraded to the specified versions. </details> #### When a new dependency is added to a runtime crate, running `syncAwsSdkLockfile` will ensure that this new dependency is included in the SDK lockfile. <details> <summary> Expand for more details...</summary> For instance, with [this hypothetical new dependency](https://github.com/smithy-lang/smithy-rs/pull/3826/files#diff-1ff3734bb74b7c43e3bd74b410f7058c6d40dbe9380458f642201035f9217457): ``` smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ sdk-lockfiles audit 2024-09-12T15:40:52.795951Z INFO sdk_lockfiles::audit: checking whether `rust-runtime/Cargo.lock` is covered by the SDK lockfile... 2024-09-12T15:40:52.827407Z INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/Cargo.lock` is covered by the SDK lockfile... 2024-09-12T15:40:52.827835Z INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/aws-config/Cargo.lock` is covered by the SDK lockfile... `jiff` (0.1.13), used by `rust-runtime/Cargo.lock`, is not contained in SDK lockfile! Error: there are lockfile audit failures ``` If we then execute ``` smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ ./gradlew aws:sdk:syncAwsSdkLockfile ... BUILD SUCCESSFUL in 1m 17s ``` the SDK lockfile will be updated to reflect only the change from `rust-runtime/Cargo.lock`: ``` smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ git diff aws/sdk/Cargo.lock diff --git a/aws/sdk/Cargo.lock b/aws/sdk/Cargo.lock index bc3870e20..c52040432 100644 --- a/aws/sdk/Cargo.lock +++ b/aws/sdk/Cargo.lock @@ -1627,6 +1627,7 @@ dependencies = [ "aws-smithy-types 1.2.6", "chrono", "futures-core", + "jiff", "time", ] @@ -2895,6 +2896,12 @@ version = "1.0.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "49f1f14873335454500d59611f1cf4a4b0f786f9ac11f4312a78e4cf2566695b" +[[package]] +name = "jiff" +version = "0.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a45489186a6123c128fdf6016183fcfab7113e1820eb813127e036e287233fb" + [[package]] name = "jobserver" version = "0.1.32" (END) ``` The updated SDK lockfile should now be in sync with runtime crates: ``` ➜ smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ sdk-lockfiles audit 2024-09-12T15:41:28.004702Z INFO sdk_lockfiles::audit: checking whether `rust-runtime/Cargo.lock` is covered by the SDK lockfile... 2024-09-12T15:41:28.034118Z INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/Cargo.lock` is covered by the SDK lockfile... 2024-09-12T15:41:28.034555Z INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/aws-config/Cargo.lock` is covered by the SDK lockfile... SUCCESS ``` </details> ---- _By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice._
Loading
Please register or sign in to comment