diff --git a/design/src/docs/code_generation.md b/design/src/docs/code_generation.md new file mode 100644 index 0000000000000000000000000000000000000000..f0a8bd3793cdb2083aff58a8c957ba4093601901 --- /dev/null +++ b/design/src/docs/code_generation.md @@ -0,0 +1,100 @@ +## Generating common service code + +How a service is constructed and how to plug in new business logic is described in [Pokémon Service][1]. +This document introduces the project and how code is being generated. It is written for developers who want to start contributing to `smithy-rs`. + +### Folder structure + +The project is divided in: + +- `/codegen`: it contains shared code for both client and server, but only generates a client +- `/codegen-server`: server only. This project started with `codegen` to generate a client, but client and server share common code; that code lives in `codegen`, which `codegen-server` depends on +- `/aws`: the AWS Rust SDK, it deals with AWS services specifically. The folder structure reflects the project's, with the `rust-runtime` and the `codegen` +- `/rust-runtime`: the generated client and server crates may depend on crates in this folder. Crates here are not code generated. The only crate that is not published is `inlineable`, +which contains common functions used by other crates, [copied into][2] the source crate + +`/rust-runtime` crates ("runtime crates") are added to a crate's dependency only when used. If a model uses event streams, it will depend on [`aws-smithy-eventstream`][3]. + +### Generating code + +`smithy-rs`'s entry points are Smithy code-generation plugins, and is not a command. One entry point is in [RustCodegenPlugin::execute][4] and +inherits from `SmithyBuildPlugin` in [smithy-build][5]. Code generation is in Kotlin and shared common, non-Rust specific code with the [`smithy` Java repository][6]. They plug into the [Smithy gradle][7] plugin, which is a gradle plugin. + +The comment at the beginning of `execute` describes what a `Decorator` is and uses the following terms: + +- Context: contains the model being generated, projection and settings for the build +- Decorator: (also referred to as customizations) customizes how code is being generated. AWS services are required to sign with the SigV4 protocol, and [a decorator][8] adds Rust code to sign requests and responses. + Decorators are applied in reverse order of being added and have a priority order. +- Writer: creates files and adds content; it supports templating, using `#` for substitutions +- Location: the file where a symbol will be written to + +The only task of a `RustCodegenPlugin` is to construct a `CodegenVisitor` and call its [execute()][9] method. + +`CodegenVisitor::execute()` is given a `Context` and decorators, and calls a [CodegenVisitor][10]. + +CodegenVisitor, RustCodegenPlugin, and wherever there are different implementations between client and server, such as in generating error types, +have corresponding server versions. + +Objects used throughout code generation are: + +- Symbol: a node in a graph, an abstraction that represents the qualified name of a type; symbols reference and depend on other symbols, and have some common properties among languages (such as a namespace or a definition file). For Rust, we add properties to include more metadata about a symbol, such as its [type][11] +- [RustType][12]: `Option`, `HashMap`, ... along with their namespaces of origin such as `std::collections` +- [RuntimeType][13]: the information to locate a type, plus the crates it depends on +- [ShapeId][14]: an immutable object that identifies a `Shape` + +Useful conversions are: + +```kotlin +SymbolProvider.toSymbol(shape) +``` + +where `SymbolProvider` constructs symbols for shapes. Some symbols require to create other symbols and types; +[event streams][15] and [other streaming shapes][16] are an example. +Symbol providers are all [applied][17] in order; if a shape uses a reserved keyword in Rust, its name is converted to a new name by a [symbol provider][18], +and all other providers will work with this [new][19] symbol. + +```kotlin +Model.expectShape(shapeId) +``` + +Each model has a `shapeId` to `shape` map; this method returns the shape associated with this shapeId. + +Some objects implement a `transform` [method][20] that only change the input model, so that code generation will work on that new model. This is used to, for example, add a trait to a shape. + +`CodegenVisitor` is a `ShapeVisitor`. For all services in the input model, shapes are [converted into Rust][21]; +[here][22] is how a service is constructed, +[here][23] a structure and so on. + +Code generation flows from writer to files and entities are (mostly) generated only on a [need-by-need basis][24]. +The complete result is a [Rust crate][25], +in which all dependencies are written into their modules and `lib.rs` is generated ([here][26]). +`execute()` ends by running [cargo fmt][27], +to avoid having to format correctly Rust in `Writer`s and to be sure the generated code follows the styling rules. + +[1]: ./pokemon_service.md +[2]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/rustlang/CargoDependency.kt#L95-L95 +[3]: https://docs.rs/aws-smithy-eventstream +[4]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/RustCodegenPlugin.kt#L34 +[5]: https://github.com/awslabs/smithy/tree/main/smithy-build +[6]: https://github.com/awslabs/smithy +[7]: https://awslabs.github.io/smithy/1.0/guides/building-models/gradle-plugin.html +[8]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/aws/sdk-codegen/src/main/kotlin/software/amazon/smithy/rustsdk/SigV4SigningDecorator.kt#L45 +[9]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/CodegenVisitor.kt#L115-L115 +[10]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/CodegenVisitor.kt#L44 +[11]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/SymbolVisitor.kt#L363-L363 +[12]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/rustlang/RustTypes.kt#L25-L25 +[13]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/RuntimeTypes.kt#L113-L113 +[14]: https://awslabs.github.io/smithy/1.0/spec/core/model.html#shape-id +[15]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/EventStreamSymbolProvider.kt#L65-L65 +[16]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/StreamingTraitSymbolProvider.kt#L26-L26 +[17]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/RustCodegenPlugin.kt#L62-L62 +[18]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/rustlang/RustReservedWords.kt#L26-L26 +[19]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/EventStreamSymbolProvider.kt#L38-L38 +[20]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/transformers/OperationNormalizer.kt#L52-L52 +[21]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/CodegenVisitor.kt#L119-L119 +[22]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/CodegenVisitor.kt#L150-L150 +[23]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/CodegenVisitor.kt#L172-L172 +[24]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/CodegenDelegator.kt#L119-L126 +[25]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/CodegenDelegator.kt#L42-L42 +[26]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/CodegenDelegator.kt#L96-L107 +[27]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/CodegenVisitor.kt#L133-L133 diff --git a/design/src/docs/pokemon_service.md b/design/src/docs/pokemon_service.md new file mode 100644 index 0000000000000000000000000000000000000000..d1209696b6d05b0cdde69948110ea4402f2f6cc2 --- /dev/null +++ b/design/src/docs/pokemon_service.md @@ -0,0 +1,236 @@ +## Code generating the Pokémon Service + +This is an overview of client and server of the Pokémon service. It introduces: + +- How a smithy-rs server customer uses the vanilla SDK and writes their business logic +- What the runtime is and how code is generated +- The folder structure of the project + +All the code shown and linked to is from the repository at this commit: [db48039065bec890ef387385773b37154b555b14][1] + +The Smithy model used to generate the code snippets is: [Pokémon][2] + +### Building the service + +The entry point of a service is [main.rs][3] + +The `PokemonService` service in the `pokemon.smithy` has these operations and resources: + +```smithy +resources: [PokemonSpecies, Storage], +operations: [GetServerStatistics, EmptyOperation, CapturePokemonOperation, HealthCheckOperation], +``` + +The entry app is constructed as: + +```rust +let app: Router = OperationRegistryBuilder::default() +``` + +`OperationRegistryBuilder` is a struct, generated [here][4], +used by service implementors to register, for each operation, the operation's implementation logic, input and output. + +```rust +pub struct OperationRegistry { + capture_pokemon_operation: Op0, + empty_operation: Op1, + get_pokemon_species: Op2, + get_server_statistics: Op3, + get_storage: Op4, + health_check_operation: Op5, + _phantom: std::marker::PhantomData<(B, In0, In1, In2, In3, In4, In5)>, +} +``` + +The builder is constructed by a `OperationRegistryBuilder`; if an operation is not passed to the builder, it will return an error. + +```rust +let app: Router = OperationRegistryBuilder::default() + .get_pokemon_species(get_pokemon_species) + .get_storage(get_storage) + .get_server_statistics(get_server_statistics) + .capture_pokemon_operation(capture_pokemon) + .empty_operation(empty_operation) + .health_check_operation(health_check_operation) + .build() + .expect("Unable to build operation registry") + .into(); +``` + +Each of these operations is a function that can take any of these signatures. + +1. If the operation is not fallible and does not share any state: + +```rust +pub async fn health_check_operation(_input: input::HealthCheckOperationInput) -> output::HealthCheckOperationOutput {...} +``` + +2. If the operation is fallible and does not share any state: + +```rust +pub async fn capture_pokemon( + mut input: input::CapturePokemonOperationInput, +) -> Result {...} +``` + +3. If the operation is not fallible and shares some state: + +```rust +pub async fn get_server_statistics( + _input: input::GetServerStatisticsInput, + state: Extension>, +) -> output::GetServerStatisticsOutput {...} +``` + +4. If the operation is fallible and shares some state: + +```rust +pub async fn get_storage( + input: input::GetStorageInput, + _state: Extension>, +) -> Result {...} +``` + +All of these are operations which implementors define; they are the business logic of the application. The rest is code generated. + +The `OperationRegistry` builds into a `Router` (`let app: Router = OperationRegistryBuilder...build().into()`). +The implementation is code generated [here][5]. + +```rust +impl + std::convert::From< + OperationRegistry, + > for aws_smithy_http_server::routing::Router +where + B: Send + 'static, + Op0: crate::server_operation_handler_trait::Handler< + B, + In0, + crate::input::CapturePokemonOperationInput, + >, + In0: 'static + Send, + ... for all Op, In +{ + fn from( + registry: OperationRegistry, + ) -> Self {...} +} +``` + +For each operation, it registers a route; the specifics depend on the [protocol][6]. +The PokemonService uses [restJson1][7] as its protocol, an operation like the `HealthCheckOperation` will be rendered as: + +```rust +let capture_pokemon_operation_request_spec = aws_smithy_http_server::routing::request_spec::RequestSpec::new( + http::Method::POST, + aws_smithy_http_server::routing::request_spec::UriSpec::new( + aws_smithy_http_server::routing::request_spec::PathAndQuerySpec::new( + aws_smithy_http_server::routing::request_spec::PathSpec::from_vector_unchecked(vec![ + aws_smithy_http_server::routing::request_spec::PathSegment::Literal(String::from("capture-pokemon-event")), + aws_smithy_http_server::routing::request_spec::PathSegment::Label, + ]), + aws_smithy_http_server::routing::request_spec::QuerySpec::from_vector_unchecked(vec![]))),); +``` + +because the URI is `/capture-pokemon-event/{region}`, with method `POST` and `region` a `Label` (then passed to the operation with its `CapturePokemonOperationInput` input struct). + +Finally, it creates a RestJSON `Router`, because that is the service's protocol. +You will have noticed, each operation is implemented as a `pub async fn`. Each operation is wrapped into an `OperationHandler`, generated [here][8]. +`OperationHandler` implements tower's `Service` [trait][9]. Implementing `Service` means that +the business logic is written as protocol-agnostic and clients request a service by calling into them, similar to an RPC call. + +```rust +aws_smithy_http_server::routing::Router::new_rest_json_router(vec![ + { + let svc = crate::server_operation_handler_trait::operation( + registry.capture_pokemon_operation, + ); +``` + +At this level, logging might be prohibited by the [`@sensitive`][10] trait. If there are no `@sensitive` shapes, the generated code looks like: + +```rust +let request_fmt = aws_smithy_http_server::logging::sensitivity::RequestFmt::new(); +let response_fmt = aws_smithy_http_server::logging::sensitivity::ResponseFmt::new(); +let svc = aws_smithy_http_server::logging::InstrumentOperation::new( + svc, + "capture_pokemon_operation", +) +.request_fmt(request_fmt) +.response_fmt(response_fmt); +``` + +Accessing the Pokédex is modeled as a restricted operation: a passcode is needed by the Pokémon trainer. +To not log the passcode, the code will be generated [here][11] as: + +```rust +let request_fmt = aws_smithy_http_server::logging::sensitivity::RequestFmt::new() + .header(|name: &http::header::HeaderName| { + #[allow(unused_variables)] + let name = name.as_str(); + let name_match = matches!(name, "passcode"); + let key_suffix = None; + let value = name_match; + aws_smithy_http_server::logging::sensitivity::headers::HeaderMarker { + value, + key_suffix, + } + }) + .label(|index: usize| matches!(index, 1)); +let response_fmt = aws_smithy_http_server::logging::sensitivity::ResponseFmt::new(); +``` + +Each route is a pair, [`BoxCloneService`][12] wrapping the service operation (the implementation) and +the information to consume the service operation. + +```rust +(tower::util::BoxCloneService::new(svc), capture_pokemon_operation_request_spec) +``` + +Now the `Router` is built. `Router` is not code generated, it instead lives in the [`aws-smithy-http-server`][13] crate. +We write Rust code in the runtime to: + +- Aid development of the project +- Have service-specific code that the majority of services share in the runtime + In Kotlin we generate code that is service-specific. + +A `Router` is a [`tower::Service`][9] that routes requests to the implemented services; hence it [implements][14] `Service` +like the other operations. + +The `Router` [implements][15] +the `tower::Layer` trait. Middleware are added as layers. The Pokémon example uses them: + +```rust +let shared_state = Arc::new(State::default()); +let app = app.layer( + ServiceBuilder::new() + .layer(TraceLayer::new_for_http()) + .layer(AddExtensionLayer::new(shared_state)), +); +``` + +The service is run by a [Hyper server][16]: + +```rust +hyper::Server::bind(&bind).serve(app.into_make_service()); +``` + +Generation of objects common to services, such as shapes, is described in [Code Generation][17]. + +[1]: https://github.com/awslabs/smithy-rs/tree/db48039065bec890ef387385773b37154b555b14 +[2]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen-server-test/model/pokemon.smithy +[3]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/rust-runtime/aws-smithy-http-server/examples/pokemon-service/src/main.rs#L34 +[4]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen-server/src/main/kotlin/software/amazon/smithy/rust/codegen/server/smithy/generators/ServerOperationRegistryGenerator.kt#L1 +[5]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen-server/src/main/kotlin/software/amazon/smithy/rust/codegen/server/smithy/generators/ServerOperationRegistryGenerator.kt#L285 +[6]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/protocols/Protocol.kt#L81 +[7]: https://awslabs.github.io/smithy/1.0/spec/aws/aws-restjson1-protocol.html +[8]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen-server/src/main/kotlin/software/amazon/smithy/rust/codegen/server/smithy/generators/ServerOperationHandlerGenerator.kt#L30 +[9]: https://docs.rs/tower-service/latest/tower_service/trait.Service.html +[10]: https://awslabs.github.io/smithy/1.0/spec/core/documentation-traits.html#sensitive-trait +[11]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/codegen-server/src/main/kotlin/software/amazon/smithy/rust/codegen/server/smithy/generators/ServerHttpSensitivityGenerator.kt#L58 +[12]: https://docs.rs/tower/latest/tower/util/struct.BoxCloneService.html +[13]: https://docs.rs/aws-smithy-http-server/latest/aws_smithy_http_server/ +[14]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/rust-runtime/aws-smithy-http-server/src/routing/mod.rs#L302 +[15]: https://github.com/awslabs/smithy-rs/blob/db48039065bec890ef387385773b37154b555b14/rust-runtime/aws-smithy-http-server/src/routing/mod.rs#L146 +[16]: https://docs.rs/hyper/latest/hyper/server/struct.Server.html +[17]: ./code_generation.md diff --git a/design/src/smithy/overview.md b/design/src/smithy/overview.md index e41d96a70c0ce1326d65707044ef2aa59db06eea..d7a8eee3d0e2a5798417c7b3a436583b2ef61556 100644 --- a/design/src/smithy/overview.md +++ b/design/src/smithy/overview.md @@ -7,8 +7,8 @@ Design documentation here covers both our implementation of Smithy Primitives (e Smithy introduces a few concepts that are defined here: 1. Shape: The core Smithy primitive. A smithy model is composed of nested shapes defining an API. -1. `Symbol`: A Representation of a type including namespaces & and any dependencies required to use a type. A shape can be converted into a symbol by a `SymbolVisitor`. A `SymbolVisitor` maps shapes to types in your programming language (e.g. Rust). In the Rust SDK, see [SymbolVisitor.kt](https://github.com/awslabs/smithy-rs/blob/c049a37f8cba5f9bec2e96c28db83e7efb2edc53/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/SymbolVisitor.kt). Symbol visitors are composable—many specific behaviors are mixed in via small & focused symbol providers, e.g. support for the streaming trait is mixed in separately. -2. `Writer`: Writers are code generation primitives that collect code prior to being written to a file. Writers enable language specific helpers to be added to simplify codegen for a given language. For example, `smithy-rs` adds `rustBlock` to [`RustWriter`](https://github.com/awslabs/smithy-rs/blob/908dec558e26bbae6fe4b7d9d1c221dd81699b59/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/rustlang/RustWriter.kt) to create a "Rust block" of code. +2. `Symbol`: A Representation of a type including namespaces and any dependencies required to use a type. A shape can be converted into a symbol by a `SymbolVisitor`. A `SymbolVisitor` maps shapes to types in your programming language (e.g. Rust). In the Rust SDK, see [SymbolVisitor.kt](https://github.com/awslabs/smithy-rs/blob/c049a37f8cba5f9bec2e96c28db83e7efb2edc53/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/SymbolVisitor.kt). Symbol visitors are composable—many specific behaviors are mixed in via small & focused symbol providers, e.g. support for the streaming trait is mixed in separately. +3. `Writer`: Writers are code generation primitives that collect code prior to being written to a file. Writers enable language specific helpers to be added to simplify codegen for a given language. For example, `smithy-rs` adds `rustBlock` to [`RustWriter`](https://github.com/awslabs/smithy-rs/blob/908dec558e26bbae6fe4b7d9d1c221dd81699b59/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/rustlang/RustWriter.kt) to create a "Rust block" of code. ```kotlin writer.rustBlock("struct Model") { model.fields.forEach { @@ -24,4 +24,6 @@ Smithy introduces a few concepts that are defined here: } ``` -3. Generators: A Generator, e.g. `StructureGenerator`, `UnionGenerator` generates more complex Rust code from a Smithy model. Protocol generators pull these individual tools together to generate code for an entire service / protocol. +4. Generators: A Generator, e.g. `StructureGenerator`, `UnionGenerator` generates more complex Rust code from a Smithy model. Protocol generators pull these individual tools together to generate code for an entire service / protocol. + +A developer's view of code generation can be found in [this document](../docs/code_generation.md).