diff --git a/aws/sdk/build.gradle.kts b/aws/sdk/build.gradle.kts index 36d93f40b62243a0ce3bbff1517241603635b9ca..2b9ecf94b38c7bad65515e0f7f5186c4702ef6e0 100644 --- a/aws/sdk/build.gradle.kts +++ b/aws/sdk/build.gradle.kts @@ -141,6 +141,9 @@ fun generateSmithyBuild(services: AwsServices): String { """ } +/** + * Task to generate smithyBuild.json dynamically + */ tasks.register("generateSmithyBuild") { description = "generate smithy-build.json" inputs.property("servicelist", awsServices.services.toString()) @@ -151,6 +154,7 @@ tasks.register("generateSmithyBuild") { doFirst { layout.buildDirectory.file("smithy-build.json").get().asFile.writeText(generateSmithyBuild(awsServices)) } + // TODO(https://github.com/smithy-lang/smithy-rs/issues/3599) outputs.upToDateWhen { false } } diff --git a/design/src/SUMMARY.md b/design/src/SUMMARY.md index b822f4fef441b3c699b3f64cda52c185a41dd19d..665fafd9e81ff98a0af30f5681cec8039a430f75 100644 --- a/design/src/SUMMARY.md +++ b/design/src/SUMMARY.md @@ -3,16 +3,11 @@ - [Design Overview](./overview.md) - [Tenets](./tenets.md) - [Design FAQ](./faq.md) -- [Transport](transport/overview.md) - - [HTTP Operations](transport/operation.md) - - [HTTP Middleware](transport/middleware.md) - - [TLS Connector](transport/connector.md) - [Smithy](./smithy/overview.md) - [Simple Shapes](./smithy/simple_shapes.md) - [Recursive Shapes](./smithy/recursive_shapes.md) - [Aggregate Shapes](./smithy/aggregate_shapes.md) - - [Endpoint Resolution](smithy/endpoint.md) - [Backwards Compatibility](smithy/backwards-compat.md) - [Client](./client/overview.md) diff --git a/design/src/smithy/endpoint.md b/design/src/smithy/endpoint.md deleted file mode 100644 index bef52bdfebdf283034ac46164725cc2d816d8e99..0000000000000000000000000000000000000000 --- a/design/src/smithy/endpoint.md +++ /dev/null @@ -1,51 +0,0 @@ -# Endpoint Resolution - -## Requirements -The core codegen generates HTTP requests that do not contain an authority, scheme or post. These properties must be set later based on configuration. Existing AWS services have a number of requirements that increase the complexity: - -1. Endpoints must support manual configuration by end users: -```rust,ignore -let config = dynamodb::Config::builder() - .endpoint(StaticEndpoint::for_uri("http://localhost:8000")) -``` - -When a user specifies a custom endpoint URI, _typically_ they will want to avoid having this URI mutated by other endpoint discovery machinery. - -2. Endpoints must support being customized on a per-operation basis by the endpoint trait. This will prefix the base endpoint, potentially driven by fields of the operation. [Docs](https://awslabs.github.io/smithy/1.0/spec/core/endpoint-traits.html#endpoint-trait) - -3. Endpoints must support being customized by [endpoint discovery](https://awslabs.github.io/smithy/1.0/spec/aws/aws-core.html#client-endpoint-discovery). A request, customized by a predefined set of fields from the input operation is dispatched to a specific URI. That operation returns the endpoint that should be used. Endpoints must be cached by a cache key containing: -```markdown -(access_key_id, [all input fields], operation) -``` -Endpoints retrieved in this way specify a TTL. - -4. Endpoints must be able to customize the signing (and other phases of the operation). For example, requests sent to a global region will have a region set by the endpoint provider. - - -## Design - -Configuration objects for services _must_ contain an `Endpoint`. This endpoint may be set by a user or it will default to the `endpointPrefix` from the service definition. In the case of endpoint discovery, _this_ is the endpoint that we will start with. - -During operation construction (see [Operation Construction](../transport/operation.md#operation-construction)) an `EndpointPrefix` may be set on the property bag. The eventual endpoint middleware will search for this in the property bag and (depending on the URI mutability) utilize this prefix when setting the endpoint. - -In the case of endpoint discovery, we envision a different pattern: -```rust,ignore -// EndpointClient manages the endpoint cache -let (tx, rx) = dynamodb::EndpointClient::new(); -let client = aws_hyper::Client::new(); -// `endpoint_req` is an operation that can be dispatched to retrieve endpoints -// During operation construction, the endpoint resolver is configured to be `rx` instead static endpoint -// resolver provided by the service. -let (endpoint_req, req) = GetRecord::builder().endpoint_disco(rx).build_with_endpoint(); -// depending on the duration of endpoint expiration, this may be spawned into a separate task to continuously -// refresh endpoints. -if tx.needs(endpoint_req) { - let new_endpoint = client. - call(endpoint_req) - .await; - tx.send(new_endpoint) -} -let rsp = client.call(req).await?; -``` - -We believe that this design results in an SDK that both offers customers more control & reduces the likelihood of bugs from nested operation dispatch. Endpoint resolution is currently extremely rare in AWS services so this design may remain a prototype while we solidify other behaviors. diff --git a/design/src/smithy/overview.md b/design/src/smithy/overview.md index f894c7e9d013779d7cba67f1bc4868376a19de8c..85df2fd66b8150572eafcde1a80063b919e2846f 100644 --- a/design/src/smithy/overview.md +++ b/design/src/smithy/overview.md @@ -1,7 +1,7 @@ # Smithy The Rust SDK uses Smithy models and code generation tooling to generate an SDK. Smithy is an open source IDL (interface design language) developed by Amazon. Although the Rust SDK uses Smithy models for AWS services, smithy-rs and Smithy models in general are not AWS specific. -Design documentation here covers both our implementation of Smithy Primitives (e.g. simple shape) as well as more complex Smithy traits like [Endpoint](./endpoint.md). +Design documentation here covers both our implementation of Smithy Primitives (e.g. simple shape) as well as more complex Smithy traits like `Endpoint`. ## Internals Smithy introduces a few concepts that are defined here: diff --git a/design/src/transport/connector.md b/design/src/transport/connector.md deleted file mode 100644 index edb50d4412d9fdf75f2740478b55019c09a8fd43..0000000000000000000000000000000000000000 --- a/design/src/transport/connector.md +++ /dev/null @@ -1,7 +0,0 @@ -The Smithy client provides a default TLS connector, but a custom one can be -plugged in. `rustls` is enabled with the feature flag `rustls`. - -The client had previously (prior to version 0.56.0) supported `native-tls`. You -can continue to use a client whose TLS implementation is backed by `native-tls` -by passing in a custom connector. Check out the `custom_connectors.rs` tests in -the `pokemon-service-tls` example crate. diff --git a/design/src/transport/middleware.md b/design/src/transport/middleware.md deleted file mode 100644 index bf7a1bc7e961f603bcabb31f05a858ff528efafd..0000000000000000000000000000000000000000 --- a/design/src/transport/middleware.md +++ /dev/null @@ -1,7 +0,0 @@ -# HTTP middleware - -Signing, endpoint specification, and logging are all handled as middleware. The Rust SDK takes a minimalist approach to middleware: - -Middleware is defined as minimally as possible, then adapted into the middleware system used by the IO layer. Tower is the de facto standard for HTTP middleware in Rust—we will probably use it. But we also want to make our middleware usable for users who aren't using Tower (or if we decide to not use Tower in the long run). - -Because of this, rather than implementing all our middleware as "Tower Middleware", we implement it narrowly (e.g. as a function that operates on `operation::Request`), then define optional adapters to make our middleware tower compatible. diff --git a/design/src/transport/operation.md b/design/src/transport/operation.md deleted file mode 100644 index 9ca8a4dfc95fee4c2338e8689c1a8d3c6b60bdbc..0000000000000000000000000000000000000000 --- a/design/src/transport/operation.md +++ /dev/null @@ -1,64 +0,0 @@ -# HTTP-based Operations -The Smithy code generator for Rust (and by extension), the AWS SDK use an `Operation` abstraction to provide a unified -interface for dispatching requests. `Operation`s contain: -* A base HTTP request (with a potentially streaming body) -* A typed property bag of configuration options -* A fully generic response handler - -In the typical case, these configuration options include things like a `CredentialsProvider`, however, they can also be -full middleware layers that will get added by the dispatch stack. - -## Operation Phases -This section details the flow of a request through the SDK until a response is returned to the user. - -### Input Construction - -A customer interacts with the SDK builders to construct an input. The `build()` method on an input returns -an `Operation`. This codifies the base HTTP request & all the configuration and middleware layers required to modify and dispatch the request. - -```rust,ignore -pub struct Operation { - request: Request, - response_handler: H, - _retry_policy: R, -} - -pub struct Request { - inner: http::Request, - properties: PropertyBag, -} -``` - -For most requests, `.build()` will NOT consume the input. A user can call `.build()` multiple times to produce multiple operations from the same input. - -By using a property bag, we can define the `Operation` in Smithy core. AWS specific configuration can be added later in the stack. - -### Operation Construction -In order to construct an operation, the generated code injects appropriate middleware & configuration via the configuration property bag. It does this by reading the configuration properties out of the service -config, copying them as necessary, and loading them into the `Request`: - -```rust,ignore -// This is approximately the generated code, I've cleaned a few things up for readability. -pub fn build(self, config: &dynamodb::config::Config) -> Operation { - let op = BatchExecuteStatement::new(BatchExecuteStatementInput { - statements: self.statements, - }); - let req = op.build_http_request().map(SdkBody::from); - - let mut req = operation::Request::new(req); - let mut props = req.properties_mut(); - props.insert_signing_config(config.signing_service()); - props.insert_endpoint_resolver(config.endpoint_resolver.clone()); - Operation::new(req) -} -``` - -### Operation Dispatch and Middleware - -The Rust SDK endeavors to behave as predictably as possible. This means that if at all possible we will not dispatch extra HTTP requests during the dispatch of normal operation. Making this work is covered in more detail in the design of credentials providers & endpoint resolution. - -The upshot is that we will always prefer a design where the user has explicit control of when credentials are loaded and endpoints are resolved. This doesn't mean that users can't use easy-to-use options (We will provide an automatically refreshing credentials provider), however, the credential provider won't load requests during the dispatch of an individual request. - -## Operation Parsing and Response Loading - -The fundamental trait for HTTP-based protocols is `ParseHttpResponse` diff --git a/design/src/transport/overview.md b/design/src/transport/overview.md deleted file mode 100644 index 42bf318814105dc7b2ce2e49bd8b87349df9d065..0000000000000000000000000000000000000000 --- a/design/src/transport/overview.md +++ /dev/null @@ -1,11 +0,0 @@ -# Transport -The transport layer of smithy-rs and the Rust SDK. Our goal is support customers to bring their own HTTP stack and runtime. - -## Where we are today -`aws-hyper` assembles a middleware stack with `tower`. It provides a way to use an HTTP client other than Hyper, however, it currently has a hard dependency on Hyper & Tokio. `hyper::Body` is being used directly as the body implementation for responses. - -## Where we want to go -1. Extend `HttpService` to add a `sleep` method. This is required to enable runtimes other than Tokio to define how they should sleep. -2. Replace `hyper::Body` in responses with SDK Body. For now, SDKBody will probably privately wrap `hyper::Body`. -3. Merge `aws-hyper` into `aws-http`. Tokio becomes an optional feature—When the Tokio feature is opted out the "fast path" variants for the connection variants are `cfg`'d out. -4. By default, customers get a fully baked HTTP stack, but they can opt out of certain features and BYO implementation of `HttpService`. diff --git a/examples/README.md b/examples/README.md index d9f67006b3b470d3f212fa26f4f4b8c57eea794b..40b8ea2180eecb700652d1880a300138f111c40f 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,7 +1,8 @@ # Smithy Rust Server SDK examples -This folder contains some example services showcasing Smithy Rust Server SDK, -also known as the Rust service framework, capabilities and to run benchmarks. +This folder contains: +- example services showcasing Smithy Rust Server SDK, also known as the Rust service framework, +- benchmarking tooling Three server implementations are available: diff --git a/tools/ci-build/changelogger/README.md b/tools/ci-build/changelogger/README.md new file mode 100644 index 0000000000000000000000000000000000000000..3d08f653d8b32758ed561697b26ced8b8e0a2128 --- /dev/null +++ b/tools/ci-build/changelogger/README.md @@ -0,0 +1,14 @@ +# ChangeLogger + +The Changelogger tool generates public facing `.md` changelogs from the structured `CHANGELOG.next.toml` formats that developers modify. The changelogger runs during smithy-rs releases to generate the smithy-rs specific changelog as well as during SDK releases to generate the SDK changelog. The smithy-rs changelog generation moves the AWS changelog entries into a separate file so that they can later be consumed. + +[smithy-rs-maintainers.txt](./smithy-rs-maintainers.txt) controls the set of users that is **not** acknowledged for their contributions in changelogs. + +## Commands +### Split +Splits changelog entries into two `json` files, splitting up `aws-sdk-rust` entries from `smithy-rs` entries. This is a prestep to full changelog generation. The end result is `aws/SDK_CHANGELOG.next.json`. + +This command is invoked as part of `smithy-rs-release`. + +### Render +Render a `.md` format changelog from a structured changelog. diff --git a/tools/ci-build/crate-hasher/README.md b/tools/ci-build/crate-hasher/README.md new file mode 100644 index 0000000000000000000000000000000000000000..8723180dcb6ba149a27c947306a281327ebcceab --- /dev/null +++ b/tools/ci-build/crate-hasher/README.md @@ -0,0 +1,11 @@ +# crate-hasher +The Crate Hasher generates deterministic hashes of crates. This is used as a dependency of `publisher generate-version-manifests` to generate the `source_hash` field: + +```toml +[crates.aws-config] +category = 'AwsRuntime' +version = '0.12.0' +source_hash = '12d172094a2576e6f4d00a8ba58276c0d4abc4e241bb75f0d3de8ac3412e8e47' +``` + +Note: (@rcoh, 4/24/2024): As far as I can tell, no tooling currently relies on the `source_hash`. It seems like this could potentially be used in `runtime-versioner audit` in the future. diff --git a/tools/ci-build/difftags/README.md b/tools/ci-build/difftags/README.md index 2b74c36ebf71155ecf1c312f205ccc25d763a9ba..4927287c785d5c6bceb725f03964e826ecb43a35 100644 --- a/tools/ci-build/difftags/README.md +++ b/tools/ci-build/difftags/README.md @@ -2,3 +2,6 @@ difftags ======== Simple CLI tool to convert a unified diff file into human readable/browsable paginated HTML files. + + +This is used by the [GitHub PR Bot](../../../.github/) diff --git a/tools/ci-build/runtime-versioner/README.md b/tools/ci-build/runtime-versioner/README.md new file mode 100644 index 0000000000000000000000000000000000000000..76bd442c0af93a7fd2ca07d4d353f8f0698e0786 --- /dev/null +++ b/tools/ci-build/runtime-versioner/README.md @@ -0,0 +1,4 @@ +# runtime-versioner +Runtime versioner serves two purposes: +1. `audit` the runtime crates to ensure that if their contents have changed, the version number in the crate has been updated. This is run as part of pre-commit. +2. `patch-runtime`: Used by `check-semver-hazards` (and manually) to test a specific set of runtime crates against the generated AWS SDK. This works by utilizing [Cargo's source patching](https://doc.rust-lang.org/cargo/reference/overriding-dependencies.html). diff --git a/tools/ci-build/sdk-lints/README.md b/tools/ci-build/sdk-lints/README.md new file mode 100644 index 0000000000000000000000000000000000000000..c6f2a0037f39c2ecb1829ec3b59901083c103b2f --- /dev/null +++ b/tools/ci-build/sdk-lints/README.md @@ -0,0 +1,2 @@ +# sdk-lints +Custom lints and fixes for sdk crates. These are run as part of `pre-commit` and during CI. diff --git a/tools/ci-build/sdk-versioner/README.md b/tools/ci-build/sdk-versioner/README.md index 45247fd4c58cba0cb84079b68ed0d22b96947e03..987d5de690d64601f75621f0c759c103abe407b6 100644 --- a/tools/ci-build/sdk-versioner/README.md +++ b/tools/ci-build/sdk-versioner/README.md @@ -7,17 +7,4 @@ within that directory, and modifies dependency lines that point to the AWS Rust or its supporting crates. These dependencies can be updated to be based on file-system path, crates.io version, or both. -Example updating SDK examples to use SDK version 0.5.0 with Smithy version 0.35.0: -```bash -$ sdk-versioner \ - --sdk-version 0.5.0 \ - --smithy-version 0.35.0 \ - path/to/aws-doc-sdk-examples/rustv1 -``` - -Example updating SDK examples to refer to local generated code: -```bash -$ sdk-versioner \ - --sdk-path path/to/smithy-rs/aws/sdk/build/aws-sdk/sdk \ - path/to/aws-doc-sdk-examples/rustv1 -``` +This tool is currently used to update all the dependencies of all examples to refer to the latest SDK version. See [aws/sdk/build.gradle.kts](../../../aws/sdk/build.gradle.kts) for more context. diff --git a/tools/ci-build/smithy-rs-tool-common/README.md b/tools/ci-build/smithy-rs-tool-common/README.md new file mode 100644 index 0000000000000000000000000000000000000000..d69142e5718a436b97596d8ebddac5db5e3516ea --- /dev/null +++ b/tools/ci-build/smithy-rs-tool-common/README.md @@ -0,0 +1,2 @@ +# smithy-rs-tool-common +Common shared libraries for runtime tooling