Rust unit testing: builtin tools

Probably you already know about mdBook. It is a wonderful tool that allows us to create books and disseminate knowledge among the community. It is used by the Book, but also by many other very interesting titles, like the the Rustonomicon, the Cargo Book, the Embedded Rust Book, or the yet unfinished Asynchronous Programming in Rust book.

Unlocking the full potential of the Open Source ecosystem requires more than useful manuals and high-quality code. To truly enable others to benefit from our work, we need to make it easy for them to use and build upon our project. That's where Rust's documentation features come in.

With Rust, we can seamlessly integrate documentation into our codebase, making it simple for third-party developers to understand and leverage our implementation. This is achieved through the use of special comments, known as doc comments, which provide a structured way to document our code.

There are two types of doc comments:

//! and /*! ...*/

Used to document, inline or as a block respectively, the parent item. Typically used for module documentation.

/// and /** ...*/

Used to document, inline or as a block respectively, the item right below them.

The rustdoc tool takes these comments, that may use markdown formatting, and generates high-quality documentation that's easily accessible. But it goes a step further. If you insert code blocks within those comments, they are treated as both, documentation examples and tests. Two for the price of one. Rust gives you the most bang for your buck!

Let's try this out incrementally. First we say something about the module at the top of the supervillain.rs file.

  //! Module for supervillains and their related stuff

We also want to document our Supervillain type.

  /// Type that represents supervillains.
  pub struct Supervillain {

Well, this was easy. Now we can produce the documentation using cargo.

  cargo doc --open --no-deps

The --open option will take care of showing you the resulting documentation. And even though, we haven't added enough comments, the structure and our comments are present in the generated documentation.

We can continue by documenting the methods of the Supervillain type. The desired format for the documentation of a method is:

[short explanations of what the item does]

[code example showing how to use it]

[OPTIONAL: more explanations and code examples in case some specific cases have to be explained in detail]

So let's document the first method:

  /// Return the value of the full name as a single string.
  ///
  /// Full name is produced concatenating first name, a single space, and the last name.
  ///
  /// # Examples
  /// ```
  /// let lex = Supervillain {
  ///     first_name: "Lex".to_string(),
  ///     last_name: "Luthor".to_string(),
  /// };
  /// assert_eq!(lex.full_name(), "Lex Luthor");
  /// ```
  pub fn full_name(&self) -> String {

If we run cargo doc --no-deps, the documentation will be refreshed (the browser page has to be refreshed too), and the example will be shown. But if we run cargo test --doc this code will also be used as a test too. There is a gotcha, though. It must be a library, not a binary.

We could move our library to its own workspace, but I am going to go with a simpler option. We can have both a library and a binary in the same crate. We just need to add a source file to be able to produce the library, but it is a tiny one, that can be created with a shell command from the root of our project.

  echo "pub mod supervillain;\n#[cfg(test)]\nmod test_common;\n" > src/lib.rs

Cargo needs to know about this too to have both possibilities, so we modify the cargo manifest by adding the following lines ¹.

  [[bin]]
  name = "detestable-me"
  path = "src/main.rs"
  test = false

  [lib]
  name = "evil"
  path = "src/lib.rs"

From the command line (or your IDE), we can use cargo to execute all the tests or just the doc tests.

  cargo t # Would execute tests twice: bin and lib if bin didn't have test = false
  cargo t --lib # or --bin <bin name> to run one or the other, but no doctests
  cargo t --doc # Run doc tests only

However, it's not all sunshine and roses. Our new doc test complains that Supervillain isn't known in the scope.

Strange. It's right there…

Well, that is true, but we have to import what we use in our doctests, as we do in our tests module. We can add Supervillain to the namespace of the doctest using the name of the library that we have declared in the manifest. We add the use statement at the beginning of the code block.

  /// use evil::supervillain::Supervillain;

And if you feel that this line doesn't add value to the example in the documentation, you can hide it using a "#" right at the beginning of the line. With this change in place, the doctests will still run, but the documentation will not have that line.

  ///# use evil::supervillain::Supervillain;

We can also use attributes at the beginning of the doctest code blocks (```<attribute>), should we need them.

ignore

If the block isn't compiled nor run with the tests.

no_run

If the block has to be compiled but not run, because it might produce undesired side effects.

should_panic

If a panic is expected when the code block is run.

compile_fail

If the block is expected to fail compilation.

edition20XX

If a specific edition of Rust has to be used with the code block.

standalone_crate

If the block cannot be combined with other doc tests.

The initial implementation of doctests was to compile each one into a separate binary. That resulted into a lot of overhead, making doctests much slower than regular unit tests. So much so that in some situations it was useful to just run the unit tests using cargo t --lib. However, from the 2024 edition, rustdoc attempts to combine the doctests into a single binary, making their execution much faster.

We can use the Rust compiler to detect when our code isn't properly documented. If we add a lint check attribute at the top of the lib.rs, the compiler will alert us (warn) or fail (deny) when any documentation is missing.

My recommended approach is that you don't add any lint at first, until you get your code to a point where it is mostly documented. Then move to a warn configuration –#![warn(missing_docs)]–, to help you complete the documentation. And once (almost) all of the warnings are gone, change the attribute to deny –#![deny(missing_docs)]– so Cargo fails compilation if any changes are made that don't include the required documentation, enforcing our codebase to remain consistently well-documented. By using this incremental strategy, we can ensure that our codebase is thoroughly documented while also allowing us to make targeted improvements.

If you are using nightly, you can also have an attribute for the missing_doc_code_examples. This lint checks if a documentation block is missing a code example. There are other lints available for rustdoc. Take a look at them.

Sometimes it is useful to get more information from your tests beyond whether they passed or failed. But if you try to print information from your tests, it won't be shown when they are executed. This might be confusing, but it's actually a deliberate design choice to show a clean report to the developer, prioritizing test output quality over unnecessary verbosity.

We can verify this behavior by adding a print statement to our set_full_name() method.

  // existing let components = ...
  println!("Received {} components.", components.len());
  // existing if components.len() ...

Even though our tests execute that method twice when we run the tests with cargo t, that output is indeed "lost." Or, at least, there is no trace of it. We just get a clean report with the list of tests, their result and a summary.

We can tell Cargo to not to capture what is sent to stdout or stderr, and to show it right away. We have to run the tests with the --no-capture option. Just remember to precede the option with a double dash (--), because otherwise the argument is passed to cargo, rather than to the subcommand cargo test.

  cargo t -- --no-capture

But if you execute that command a few times, you will notice that the output of the two invocations of that print statement appear in different positions of the report. Cargo runs your tests in parallel using threads and the order of execution is not predictable. If this is confusing with two print statements, imagine the output of a full-fledged application.

However, there is another way to get this information to be displayed. We can use the option =--show-output instead. This prints the output, with other information, after all the tests have been run, but in an orderly fashion. Using this option the output of each test is more readable and it doesn't get mixed with the output of other tests. The price that we pay in exchange is that the report is more verbose.

  cargo t -- --show-output

It is reassuring to have a good test harness, but there are times when you don't want to run every single test. Perhaps you're testing a specific scenario or feature, and want to focus on some tests and ignore the others. Maybe, you might need to execute a single test in isolation. In those dark moments of testing uncertainty, you might struggle to recall the names of all your tests, and particularly the ones that you want to run.

You can use cargo with the --list option to list the available tests. Remember to precede the option with the double dash (--) as I mentioned before.

  cargo t -- --list

This produces this output (that you can narrow down using ripgrep, for example):

    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.01s
     Running unittests src/lib.rs (target/debug/deps/evil-5ab356f2245c3541)
supervillain::tests::attack_shoots_weapon: test
supervillain::tests::full_name_is_first_name_space_last_name: test
supervillain::tests::plan_is_sadly_expected: test
supervillain::tests::set_full_name_panics_with_empty_name: test
supervillain::tests::set_full_name_sets_first_and_last_names: test
supervillain::tests::try_from_str_slice_produces_error_with_less_than_two_substrings: test
supervillain::tests::try_from_str_slice_produces_supervillain_full_with_first_and_last_name: test

7 tests, 0 benchmarks
   Doc-tests evil
src/supervillain.rs - supervillain::Supervillain::full_name (line 23): test

1 test, 0 benchmarks

As you can see, you get a list with all the unit tests and it includes the doctests too. Notice there are 2 tests that start with try_from_str and 4 tests contain "full_name" in their name (including the doctest.) Let's use them to illustrate some uses.

We start by adding the #[ignore = "temporary disabled"] attribute in the second test (try_from_str_slice_produces_supervillain_full_with_first_and_last_name). Also, we change the beginning of our doctest block to /// ```ignore. If we run the tests, these two will be skipped and the report will also show the reason that we included in the unit test attribute.

test supervillain::tests::try_from_str_slice_produces_supervillain_full_with_first_and_last_name ... ignored, temporary disabled
...
test src/supervillain.rs - supervillain::Supervillain::full_name (line 23) ... ignored

We can run just the ignored tests with:

  cargo t -- --ignored

or run all the tests, including the ignored ones:

  cargo t -- --include-ignored

We could select and run the tests by name. For example, we can select all the tests that contain the string "full_name". But if we do, only the unit tests will be run.

% cargo t full_name                                                                     250918142706
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.01s
     Running unittests src/lib.rs (target/debug/deps/evil-5ab356f2245c3541)

running 3 tests
[[test supervillain::tests::set_full_name_sets_first_and_last_names ... ok
test supervillain::tests::full_name_is_first_name_space_last_name ... ok
test supervillain::tests::set_full_name_panics_with_empty_name - should panic ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 4 filtered out; finished in 0.00s

If we want the doctests to be selected by the string, we need to be a little bit more explicit with cargo t full_name --doc, but it will only pick tests from the doctests list.

We can also combine test selection with other options. For example, if we want all the tests that contain the string "from_str", ignored or not, we should use the following command.

  cargo t from_str -- --include-ignored

If we just want to select the tests for the full_name() method, we have to be more specific with the name.

  cargo t -- full_name # It also selects set_full_name() tests
  cargo t -- Supervillain::full_name # selects the doctest
  cargo t -- ::full_name # Without -- doesn't select doctest

And, if we want to just run a test that exactly matches the string, we can use the --exact option. And remember that the complete name can be obtained from the list.

  cargo t -- --exact supervillain::test::full_name_returns_first_name_space_last_name

We create tests to improve the quality of our codebase and to scare away errors. Yet, errors sneak their way into our code and into our tests.

To err is human, to debug is divine. But certainly, not obvious. An executable is produced with the tests and it isn't the same one that we create to run our application. Where is that binary then? And how do we connect to it and debug it?

Let's take it step by step. We need an error in our codebase that isn't obvious. We can add an extra space in the code that splits the full name into its components. The important difference with the existing code is that I have added a *second space to the argument of the split() method.

  fn try_from(name: &str) -> Result<Self, Self::Error> {
      let components = name.split("  ").collect::<Vec<_>>();

When we run the tests we get one of the tests failing.

% cargo t                                                                               250918164712
   Compiling detestable-me v0.1.0 (/Users/jorge/Developer/Rust/detestable-me)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.23s
     Running unittests src/lib.rs (target/debug/deps/evil-5ab356f2245c3541)

running 7 tests
...
test supervillain::tests::try_from_str_slice_produces_supervillain_full_with_first_and_last_name ... FAILED
...

The output produced when we run the tests show the name of the binary used in the process (target/debug/deps/evil-5ab356f2245c3541)². This is the one we need to use with the debugger:

  rust-lldb target/debug/deps/evil-5ab356f2245c3541

Now, we can set a breakpoint at the beginning of the failing test.

  br set -r try_from_str_slice_produces_supervillain_full_with_first_and_last_name

We run it with "r" and, if there is more that one breakpoint location, we continue "c" until you see the source. Then we get go to the next line ("n") which invokes the TryFrom associated function. We want to step into ("s") its code to see what happens there.

In the try_from() function, we execute the first line (n) and check the value of the variable components.

(lldb) p components
(alloc::vec::Vec<&str, alloc::alloc::Global>) size=1 {
  [0] = "Darth Vader" {
    [0] = 'D'
    [1] = 'a'
    [2] = 'r'
    [3] = 't'
    [4] = 'h'
    [5] = ' '
    [6] = 'V'
    [7] = 'a'
    [8] = 'd'
    [9] = 'e'
    [10] = 'r'
  }
}

This should raise our suspicions, because we have one element in the components vector, but it does have a space. Time to take a look at the argument of split(), realize that we made a mistake with the argument, fix it, run the tests again, and get a T-shirt with the message "I debugged an error that I found with my Rust tests." 😄