Skip to content
Unverified Commit 07dd8d46 authored by ysaito1001's avatar ysaito1001 Committed by GitHub
Browse files

Create CredentialsCache and DefaultResolver only once in test/bench (#2620)

## Motivation and Context
Addresses
https://github.com/awslabs/smithy-rs/pull/2593#discussion_r1170703265
and
https://github.com/awslabs/smithy-rs/pull/2593#discussion_r1170703694.

Also ports the changes made in #2592.

## Description
Prior to this PR, `sra_manual_test` and the bench
`middleware_vs_orchestrator` created `CredentialsCache` and
`aws_sdk_s3::endpoint::DefaultResolver` every time a request for
`ListObjectsV2` was dispatched. This is not the case in production where
those two types are created once during the construction of a service
config
([here](https://github.com/awslabs/aws-sdk-rust/blob/main/sdk/s3/src/config.rs#L623-L625)
and
[here](https://github.com/awslabs/aws-sdk-rust/blob/main/sdk/s3/src/config.rs#L635-L652)

)
and reused for subsequent request dispatches.

The PR will make `sra_manual_test` and `middleware_vs_orchestrator` do
the same, creating `CredentialsCache` and
`aws_sdk_s3::endpoint::DefaultResolver` only once before the
test/benchmark starts running and reusing them thereafter.

The change will help bring the performance for `orchestrator` closer to
that for `middleware`.

- In the `main` branch with `DefaultEndpointResolver` commented in and
`StaticUriEndpointResolver` commented out:
```
middleware              time:   [20.924 µs 20.943 µs 20.964 µs]
                        change: [-1.0107% -0.7357% -0.4827%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) high mild
  6 (6.00%) high severe

orchestrator            time:   [933.68 µs 940.11 µs 945.82 µs]
                        change: [+2735.7% +2754.5% +2770.9%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 17 outliers among 100 measurements (17.00%)
  14 (14.00%) low mild
  2 (2.00%) high mild
  1 (1.00%) high severe
```

- With the change in this PR:
```
middleware              time:   [21.161 µs 21.194 µs 21.232 µs]
                        change: [-0.8973% -0.6397% -0.3758%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe

orchestrator            time:   [56.038 µs 56.182 µs 56.349 µs]
                        change: [-0.7921% -0.5552% -0.3157%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

```


## Testing
Executed the following without any errors:
```
➜  smithy-rs git:(ysaito/create-creds-cache-and-ep-resolver-only-once) ✗ ./gradlew aws:sra-test:assemble
➜  aws-sdk-s3 git:(ysaito/create-creds-cache-and-ep-resolver-only-once) cargo t
➜  aws-sdk-s3 git:(ysaito/create-creds-cache-and-ep-resolver-only-once) cargo bench
```

----

_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._

---------

Co-authored-by: default avatarYuki Saito <awsaito@amazon.com>
parent 982319ff
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment