Running Rust Tasks With “xtask” and “xtaskops”

Pop quiz: How do you write and run your daily tasks?

If you have a moderately large React project or a project in Go, or any other kind of language, you probably found yourself needing your own build, tests or CI tasks.

For example, for npm or yarn in Node.js, you'd write your own test:integration task which run only your integration tests:

$ yarn test:integration

And test:integration might be any series of shell commands which tell your test runner what to filter in.

And sometimes, these jobs have dependencies, so you order them in the right way:

$ yarn clean && yarn build && yarn test:integration

And then, you drop those into your package.json file:

"ci": "yarn clean && yarn build && yarn test:integration"

Which eventually become:

$ yarn ci

For make or similar tools, for all kinds of tasks such as build, lint, test, coverage, CI, release, it looks pretty similar.

Sometimes you have to write your own “CI tooling” shell scripts:

$ ./ci/build-dev.sh

How do we do this in Rust projects?

It’s not really possible to do ad-hoc tasks with Cargo like it is with yarn or npm, by adding some shell script to your manifest file (for Rust - cargo.toml).

People usually solve this with custom shell scripts invoked for a specific need, or by using (abusing?) make, similar to how the Go community did it.

There are a few reasons why we’re going the easy route like this. They are:

• It’s easy, just name the task and go
• We all know shell commands
• Running order and failure modes are inherited from the shell, too
• Easy to experiment & iterate

The workflow problem

Trying out a shell command, and pasting it as a make task is great for simple cases.

But, for any moderately large project, ask youself these questions:

• How do you supply custom parameters to make? (or npm tasks for that matter)
• How about supplying parameters to a shell script? and flags or options? and defaults?

The answer is: you go back to what ever shell script voodoo you are familiar with at that moment.

Prompting for user input is even worse, and running tasks in a specific order (resolving a graph of tasks) is left to you, hand-coding the order.

The productivity problem

Like in “normal” software, to be productive you need to:

• Do some programmatic logic
• Reuse your existing codebase
• Import other people’s libraries or tasks
• Ability to debug, reason about, or test tasks

All of these are not possible in npm, or make or similar tools, and it's a painfully bad experience with authoring shell scripts.

What are the solutions out there?

Obviously, there are a few solutions for this challenge.

Declarative “Linear” build tools

There are your typical tools that were grown to “run my shell command” such as yarn, make, etc.

You get to use everything you know about environment variables, shell command ordering and logic ( &&, etc), use positional params $1, $2, etc. and hope for the best.

Smarter ‘make’ tools, semi-declarative

Tools like just, or cargo-make are purpose-built to solve task-related pains, which is a good thing.

For example building tasks with cargo-make looks like this:

[tasks.A]
dependencies = ["B", "C"]

[tasks.B]
dependencies = ["D"]

[tasks.C]
dependencies = ["D2"]

[tasks.D]
script = "echo hello"

[tasks.D2]
alias="D"

So we get:

• Task-first tooling with dependencies, order resolution
• Convenience for parameters and naming

But, we’re ultimately limited by running commands on the shell, or the tool we’re using.

Programmatic language DSL tooling

In this category, you have a library or an infrastructure, and you’re left to build tasks on your own.

All you get is a way to run your fully-programming-language capable tasks in a convenient way, and some glue.

A great example for this was rake in Ruby. It looked like make but you build all of your tasks in your native language: Ruby.

At some point in time, a similar solution (which is more of a guideline) emerged for Rust: xtask.

xtask

xtask is a convention, a project set up practice.

If you follow the set up steps, you end up with a cargo xtask command and a new Rust project called xtask in which you build all your custom stuff.

Generally using xtask means you use cargo as you're used to.

It also means you are able to use Rust to its full potential: import libraries, reuse project logic, test, lint and run your build tasks safely.

Setting up xtask

We’re trying to move from a single crate project to a multiple crate workspace, where you have two members:

[ your-project, xtask ]

If your project is called acme move your code to a new acme/ inner folder.

Then, within your project:

$ cargo new --bin xtask

Set up your workspace (root cargo.toml):

[workspace]
members = [
  "acme",
  "xtask"
]

Add an alias for cargo to know xtask (in .cargo/config):

[alias]
xtask = "run --package xtask --"

That’s it. Now the xtask project is yours to build. Treat it like any other binary that should perform build tasks.

This is where you’d want to:

• Take commands and parameters easily
• Route the commands to tasks your wrote easily
• Author tasks easily

Right now, there’s no recommended way to author xtask tasks easily.

This is why xtaskops was born.

Using xtaskops

Using xtaskops, here is a fully loaded xtask binary, with more tasks than you'd need. It fits a single screen more or less:

By using the xtaskops library, you get all of the common tasks in Rust for free, and then some cool ones too like bloat-deps (show biggest dependencies, by size) and powerset (runs tests over combinations of features).

Just wire them to your CLI framework of choice (here, we’re using clap).

Here’s some tasks that are available today in xtaskops:

• bloat_deps Show biggest crates in release build
• bloat_time Show crate build times
• dev Run cargo check followed by cargo test for every file change
• ci Run typical CI tasks in series: fmt, clippy, and tests
• coverage Run coverage
• docs Run cargo docs in watch mode
• install Instal cargo tools
• powerset Perform a CI build with powerset of features

For running your own commands, use the cmd! macro:

cmd!(your,command,here)

Using xtask for every project

Instead of manually setting up xtask over and over for every project, you can use rust-starter which is xtask-driven.

For existing projects, you can copy the xtask files or use Backpack which can grab parts of repositories easily.

Of course, you may want to make a starter project of your own: set up xtask once for yourself, and reuse your project for all of your work. Then use Backpack to kick off every new project from your repository address.

Conclusion

Before you add another make task, or write another custom shell build script, remember:

• Your build, ci, workflow, errands code — is still maintained code
• Don’t underestimate maintenance overhead for a task using shell commands
• Your shell voodoo better be bug free, tested, and reasoned about
• It’s better to reuse team investment in knowledge, code, standards, and practices
• make can be tricky and require forgotten knowledge (e.g. .PHONY)
• package.json tasks often require additional packages ( rimraf, concurrently)

You can use xtask to alleviate all these pains, and write more Rust code.