Skip to content
Unverified Commit 3499f60e authored by ysaito1001's avatar ysaito1001 Committed by GitHub
Browse files

Enhance gradle tasks for managing lockfiles (#3829)

## Description
This PR introduces and updates gradle tasks for managing lockfiles. Here
are the highlights:
- [The SDK
lockfile](https://github.com/smithy-lang/smithy-rs/blob/main/aws/sdk/Cargo.lock)
can now be generated directly within the `smithy-rs` repository without
the `aws-sdk-rust` repository.
- The SDK lockfile can be synchronized with runtime lockfiles, updating
only new dependencies while preserving the versions of existing ones.
- To prevent updating broken dependencies to the latest versions, we
track the last known good versions and downgrade them to those versions.

New/updated gradle tasks are intended for automation:
- This existing task no longer requires `-Paws-sdk-rust-path`. We plan
to incorporate it into a weekly GitHub Action to automate lockfile
updates:
```
./gradlew aws:sdk:cargoUpdateAllLockfiles
```
- This new task synchronizes the SDK lockfile with runtime lockfiles. We
plan to integrate it into pre-commit hooks:
```
./gradlew aws:sdk:syncAwsSdkLockfile  
```

In addition, this PR has updated the SDK lockfile by executing
`./gradlew aws:sdk:syncAwsSdkLockfile`. The updated lockfile no longer
includes many SDK crates that are unused in CI/CD processes. The new SDK
lockfile is in sync with the runtime lockfiles:
```
➜  smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) sdk-lockfiles audit
2024-09-12T16:02:25.193765Z  INFO sdk_lockfiles::audit: checking whether `rust-runtime/Cargo.lock` is covered by the SDK lockfile...
2024-09-12T16:02:25.224862Z  INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/Cargo.lock` is covered by the SDK lockfile...
2024-09-12T16:02:25.225389Z  INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/aws-config/Cargo.lock` is covered by the SDK lockfile...
SUCCESS
```

## Testing
I have verified the change against basic use cases:

#### When running `cargoUpdateAllLockfiles`, dependencies will be
updated to their latest versions, while broken crates will be pinned to
the last known good versions.

<details>
<summary> Expand for more details...</summary>

When we execute  
```
smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ ./gradlew aws:sdk:cargoUpdateAllLockfiles
...
BUILD SUCCESSFUL in 1m 7s
```
all lockfiles include the latest versions of dependencies, except for
those that are pinned due to being broken. Currently, minicbor is
[pinned to
0.24.2](https://github.com/smithy-lang/smithy-rs/blob/7f1d992214c25347fffeea63f8ceb58f77c7a8bb/aws/sdk/build.gradle.kts#L503-L504):
```
➜  smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ git status
On branch ysaito/enhance-gradle-tasks-for-lockfile
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   aws/rust-runtime/Cargo.lock
	modified:   aws/rust-runtime/aws-config/Cargo.lock
	modified:   aws/sdk/Cargo.lock
	modified:   rust-runtime/Cargo.lock

no changes added to commit (use "git add" and/or "git commit -a")
```
```
➜  smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ cat aws/sdk/Cargo.lock | rg -C1 minicbor
...
---
[[package]]
name = "minicbor"
version = "0.24.2"
---
...

➜  smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ cat rust-runtime/Cargo.lock | rg -C1 minicbor
...
---
[[package]]
name = "minicbor"
version = "0.24.2"
---
...
```
Finally, the `sdk-lockfiles audit` command should run successfully after
updating all lockfiles:
```
➜  smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ sdk-lockfiles audit                       
2024-09-12T15:35:47.890530Z  INFO sdk_lockfiles::audit: checking whether `rust-runtime/Cargo.lock` is covered by the SDK lockfile...
2024-09-12T15:35:47.922468Z  INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/Cargo.lock` is covered by the SDK lockfile...
2024-09-12T15:35:47.922898Z  INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/aws-config/Cargo.lock` is covered by the SDK lockfile...
SUCCESS
```

I also specified multiple broken dependencies and verified they were all
downgraded to the specified versions.

</details>

#### When a new dependency is added to a runtime crate, running
`syncAwsSdkLockfile` will ensure that this new dependency is included in
the SDK lockfile.

<details>
<summary> Expand for more details...</summary>

For instance, with [this hypothetical new
dependency](https://github.com/smithy-lang/smithy-rs/pull/3826/files#diff-1ff3734bb74b7c43e3bd74b410f7058c6d40dbe9380458f642201035f9217457):
```
smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ sdk-lockfiles audit
2024-09-12T15:40:52.795951Z  INFO sdk_lockfiles::audit: checking whether `rust-runtime/Cargo.lock` is covered by the SDK lockfile...
2024-09-12T15:40:52.827407Z  INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/Cargo.lock` is covered by the SDK lockfile...
2024-09-12T15:40:52.827835Z  INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/aws-config/Cargo.lock` is covered by the SDK lockfile...
`jiff` (0.1.13), used by `rust-runtime/Cargo.lock`, is not contained in SDK lockfile!
Error: there are lockfile audit failures
```
If we then execute  
```
smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ ./gradlew aws:sdk:syncAwsSdkLockfile
...
BUILD SUCCESSFUL in 1m 17s
```
the SDK lockfile will be updated to reflect only the change from
`rust-runtime/Cargo.lock`:
```
smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ git diff aws/sdk/Cargo.lock 
diff --git a/aws/sdk/Cargo.lock b/aws/sdk/Cargo.lock
index bc3870e20..c52040432 100644
--- a/aws/sdk/Cargo.lock
+++ b/aws/sdk/Cargo.lock
@@ -1627,6 +1627,7 @@ dependencies = [
  "aws-smithy-types 1.2.6",
  "chrono",
  "futures-core",
+ "jiff",
  "time",
 ]
 
@@ -2895,6 +2896,12 @@ version = "1.0.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "49f1f14873335454500d59611f1cf4a4b0f786f9ac11f4312a78e4cf2566695b"
 
+[[package]]
+name = "jiff"
+version = "0.1.13"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8a45489186a6123c128fdf6016183fcfab7113e1820eb813127e036e287233fb"
+
 [[package]]
 name = "jobserver"
 version = "0.1.32"
(END)
```
The updated SDK lockfile should now be in sync with runtime crates:
```
➜  smithy-rs git:(ysaito/enhance-gradle-tasks-for-lockfile) ✗ sdk-lockfiles audit
2024-09-12T15:41:28.004702Z  INFO sdk_lockfiles::audit: checking whether `rust-runtime/Cargo.lock` is covered by the SDK lockfile...
2024-09-12T15:41:28.034118Z  INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/Cargo.lock` is covered by the SDK lockfile...
2024-09-12T15:41:28.034555Z  INFO sdk_lockfiles::audit: checking whether `aws/rust-runtime/aws-config/Cargo.lock` is covered by the SDK lockfile...
SUCCESS
```
</details>

----

_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
parent db1a9f19
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment