How to do code coverage in Rust
- History of code coverage in Rust
- Tarpaulin
- gcov
- Source based coverage
- Our goals
- Running Locally
- Automating with xtask
- Inline coverage with VSCode
- Running in your CI
Now that we have source based coverage in Rust, weâre going to look at how to do the following:
- Run coverage locally
- Integrate with IDE for inline coverage indicators
- Run coverage in CI and upload results to a coverage service provider
But first, letâs look at the history of code coverage in Rust, it might help you when you Google around for solutions, to understand whatâs current and whatâs not.
History of code coverage in Rust
Tarpaulin
Tarpaulin is an easy tool to run code coverage with Rust, but has limitations. An intuitive description for how Tarpaulin works: it will instrument your code and then use ptrace
to eavesdrop on what's happening in order to count lines for coverage analysis.
This is why Tarpaulin only supports Linux, and also specifically only supports x86_64 processors.
For accuracyâs sakeâââTarpaulin provides line coverage and is fairly reliable but still include some inaccuracies in the results.
gcov
Before source based coverage, Rust relied on the gcov
technique, which relied on debug info to map from LLVM IR (bottom, generated and hidden from you) to lines of source code (top, what you're used to seeing). Think about it like making a really good guess for names into lines of code, and keeping count of which line is ran and how many times.
As such, and since the resolution of information is lost from layer to layer as you move from source to compiled binary form, the gcov
based techinque was not the most accurate.
Source based coverage
Source based coverage lets Rust have instrumentation done right at the compiler level while then generating IR from source, which moves down to all compilation stages, and then lets the coverage analysis tools use all of that high-quality information to correlate and pinpoint exactly a given low-level construct to its higher-level source code (that you wrote).
This results in a much higher accuracy level and better implementation of coverage instrumentation. To get a feel of what âaccuracy isâ, then take a look at this short-circuit statement:
if a < b && c > d {
// ..
}
If a > b
(the inverse of the first condition), we know that the next condition does not need to be run. This is shortcircuiting. And so, source based coverage will be able to tell you that c > d
was never exercised, so you could write a test for that condition as well, where the gcov method would list all of that line as exercised (!).
Today, source based coverage in Rust is the recommended way to do coverage. Itâs also the most usable, most accurate, and most portable way to do coverage.
Our goals
Source based coverage opens up the opportunity to have a holy-trinity of coverage:
- Run locally for fast feedback for an individual.
- Seamless inline IDE integration
- Run in CI, PRs, and centrally for a team
Running Locally
To do code coverage with the new source based coverage features, you need to perform a few moves. Weâll see the manual steps first, and automate it all later.
Generally we want tests to drive our coverage, so weâll run cargo test
, but we need to enable instrumentation, and also tell LLVM to output its raw profiling data.
$ CARGO_INCREMENTAL=0 RUSTFLAGS='-Cinstrument-coverage' LLVM_PROFILE_FILE='cargo-test-%p-%m.profraw' cargo test
Next, we need to take all of that data, and generate a report that can be ingested by tools and read by machines, or that we can browse around as humans.
As HTML:
$ grcov . --binary-path ./target/debug/deps/ -s . -t html --branch --ignore-not-existing --ignore '../*' --ignore "/*" -o target/coverage/html
As lcov
, a popular output format:
$ grcov . --binary-path ./target/debug/deps/ -s . -t lcov --branch --ignore-not-existing --ignore '../*' --ignore "/*" -o target/coverage/tests.lcov
Then, itâs up to us what to do with the data:
- Tell our IDE to find the
lcov
files for inline coverage highlights - Upload the
lcov
files to code coverage providers such as Codecov - Use the
lcov
data to create custom reports or to fail tests
When done, weâll need to clean up the *.profraw
files that were littered alongside our code.
Automating with xtask
cargo-xtask
is the current implementation of cargo tasks. Think of it as a Rust-based alternative to make
, but that it is pure Rust, all around -- it's just another project in your workspace, and there are very little strict rules around it other than a few best practices. To read more about it check out the xtask repo
Hereâs how all of this process looks with my current flavor of cargo xtask cover
(you can take a look at the full example in my rust-starter):
Now if you run:
$ cargo xtask coverage
Youâll have lcov
files waiting for you in coverage/
which you can upload to a coverage provider or use in other tools.
And if you run:
$ cargo xtask coverage --dev
Youâll be offered to open a report which you can load in your browser, no other coverage product or provider needed.
Inline coverage with VSCode
If you want to see coverage inline with your code, as you type, you can do that with the coverage gutters extension, which is recommended to use with any language that can produce lcov
really, so not only useful for Rust.
After youâve installed it, you need to let it know where to find lcov
files generically in your project.
For us, if you use the xtask
above for all Rust projects, it will always be in: coverage/tests.lcov
.
So this is the configuration I have, as an example:
"coverage-gutters.coverageFileNames": [
"coverage/tests.lcov",
"lcov.info",
"cov.xml",
"coverage.xml",
"jacoco.xml",
"coverage.cobertura.xml"
],
You can then view your inline coverage with watch or just toggle it:
Running in your CI
Next, we want to use the same coverage task in our CI, which is another benefit we get by using xtask
. Basically, running our coverage in CI will reduce to:
$ cargo xtask coverage
And finding a way to upload coverage/*.lcov
to our coverage provider.
In this example, weâll use Github Action as our CI, and Codecov as our provider, but it shouldnât be too different for any other set of vendors.
Our Github workflow should also include set up for the various coverage tools we need, so first install llvm-tools-preview
.
- name: Install toolchain
id: toolchain
uses: actions-rs/toolchain@v1
with:
toolchain: stable
target: x86_64-unknown-linux-musl
override: true
components: llvm-tools-preview
Then, we download grcov
, work through the rest of our task, and upload to Codecov:
- name: Download grcov
run: |
mkdir -p "${HOME}/.local/bin"
curl -sL https://github.com/mozilla/grcov/releases/download/v0.8.10/grcov-x86_64-unknown-linux-gnu.tar.bz2 | tar jxf - -C "${HOME}/.local/bin"
echo "$HOME/.local/bin" >> $GITHUB_PATH
- name: Run xtask coverage
uses: actions-rs/cargo@v1
with:
command: xtask
args: coverage
- name: Upload to codecov.io
uses: codecov/codecov-action@v3
with:
files: coverage/*.lcov
Thatâs it, at this point you should have:
- Local coverage tools and task for fast feedback
- Smooth and seamless IDE integration
- CI integration with a nice code coverage provider for working in a team and automatically covering PRs
You can find the full workflow in the rust-starter project.
You can also use the project itself as a starting point to get all of those benefits before writing a single line of code.Originally published at https://blog.rng0.io .