How to easily implement a configuration-first provider pattern in Rust 🦀
- The “registry”
- Serde can deserialize traits with typetag
- Nested blocks and nested configuration
- Self-building blocks
In this post we’ll review a way to build a provider pattern in Rust which is dynamic, extensible, and fun to maintain, while as statically typed as possible.
You might need a provider pattern when building a configuration for a system that has plugins, or different swappable components, logic encoded with no-code approaches, workflow configuration (think: CI/CD YAMLs), and more.
Here’s an example YAML of two “blocks” for some kind of a workflow runner. It lets you configure which blocks run one after another and their settings.
Below, we see an
env_block setting environment variables and a
shell_block running a shell command. Each block is configured via the same YAML, and, once loaded, has a single
- kind: env_block
- kind: shell_block
cmd: "echo $PATH"
We’ll implement dynamic block creation from configuration, or as some would say plugin system, or registry of blocks, or strategy design pattern, or provider pattern.
Ultimately we want something like this:
let blocks_config = load_yaml; // box dyn for fexibility
// do something to turn config into Vec<Box<dyn Block>>
blocks = figure_it_out;
This code is completely ignorant of the different types of blocks we have, or their different settings.
Remember: we have a collection of
block s. Each block, has a different concrete implementation, and each block can have its own configuration:
env_blocksets a bunch of environment variables
shell_blockruns a shell command
- any other block..
Each block has a
run function, available through a
Normally we would need something that knows about all different kinds of blocks, so maybe something like a blocks registry,
// load YAML
Some issues with this:
- Late binding: we only know what the rest of the configuration fields mean after we deal with the
.kindattribute, so we'll have to keep it as
- Or, we’ll have to think of a different kind of configuration format which will deserialize into a strongly typed model, such as:
Keep in mind that the above solution loses ordering between
shell_block. There are additional solutions, but they revolve around the same idea: encoding types in configuration structure so that deserialization has enough power to infer those types.
What ever solution we’ll find, it’s not as easy to maintain — when we need to add new kinds of blocks, when some blocks are enabled only by features, or when we have complex block creation logic.
Serde can deserialize traits with
What if we can deserialize each block options into some behavior that can build a block from the options?
The key here is typetag.
typetag is a great library that you can use to have
serde deserialize trait objects using a single discriminating field for indicating a type, ultimately "hidden" by a trait.
And once we can deserialize trait objects, we can dial into that dynamic behavior that we wanted.
Below, our type tag in the YAML is
kind. This is all the information that's needed to decide which implementation of
block the next sibling fields in the YAML relate to.
- kind: env_block
- kind: shell_block
cmd: "echo $PATH"
Now, we define a
BlockBuilder trait. We say that a
BlockBuilder's job is to take a particular block's YAML options and configure a
block for us based on those options, remember, it returns a
Box<dyn Block> because it needs to build any kind of block for us, and behind the scenes it will also magically pick the concrete type for us.
The important bit is the
tag = "kind" attribute, which tells
typetag to generate the right code to build a specific
BlockBuilder that relates to our specific
Why a seperate abstraction for building our block? because we recognize that a “live” block, may contain fields that are initialized but not serializable, for example a live database connection, while a block configuration section may contain a connection string . This way, as a best-practice, we separate the behavior of constructing blocks from configuration from the actual block. We’ll show an example later where we can combine the two.
Here’s the data model for our env block options on the Rust side, as defined in the YAML file:
We see that it’s an
env block, which is one kind of block our of all possible blocks we have.
We implement a
BlockBuilder, and note the specific values we set for the
// The actual env block (which isn't so important for the purpose of this discussion)
Now we can load all block configurators:
// `load_yaml` just reads and deserializes yaml, no special code
let block_configs = load_yaml; // Vec<Box<dyn BlockBuilder>>
// generically call builders to get all needed blocks.
// we don't know or care about specific block implementations,
// we're getting a Box<dyn Block> which has a `.run` function and that's perfect for us.
let blocks = block_configs.iter.for_each.;
Note that there nothing in the code above indicating a specific kind of block or a specific implementation of a block. Everything is fully dynamic and extensible, and our code is ignorant of how many types of blocks there are, how to build them, and so on.
To illustate what’s happening when building an
-> we deserialize YAML
-> uses `kind=env_block` to resolve EnvBlockOpts
-> deserialized EnvBlockOpts into a <dyn BlockBuilder> trait which exposes a `build` function
-> we call `.build`
-> get a fully configured <dyn Block>
Nested blocks and nested configuration
What if we have a block that contains a block, and nests its YAML configuration as well?
# a top level ping block
- kind: ping_block
# billing block
# auth block
.run calls both children blocks's
.run in order to accomplish what it needs to do.
The full implementation of dynamically loading
ping_block would be:
// note: billing and auth config builders will be automatically resolved by typetag but
// `.build` will not be automatically called for the nested blocks
// this is our actual ping block, which is composed out of two other blocks
// individual block builders, with their appropriate opts:
// The key is implementing a nested `build` in the top level block builder,
// which calls individual nested block builders manually
Sometimes a block can have serializable fields and its construction is very simple.
In that case we can save a bit of typing and have a bloc implement both
BlockBuilder and the
Block trait so it can build itself with no need for a seperate "opts" struct.
// the block struct itself is serializable
// note we implement a config builder for HealthBlock itself, there's no separate configuration opts struct
// and now:
let builder = load_yaml; // this is the config builder trait
let block = builder.build; // which builds itself
This is a compact way to implement this pattern, and you can start with it by default.