PyTest - A Journey Of 3 Years

15th Aug 2020
6 min read
Tags:
Pytest

Salt is ending a journey that started 3 years ago with the first pull request to add support for PyTest. In the upcoming weeks, Salt will finally switch to PyTest as its sole test runner, and the previous, and heavily customized unittest runner, runtests.py, finally reaches EOL.

Why?

runtests.py served us well, but every time the Salt test suite got a new sub-directory, runtests.py had to be updated in order to pick up those new tests. Often enough, those tests, or sub-directories, would be added and never actually run on the CI pipelines because the needed updates to runtests.py were lacking.

When we needed additional daemons running for the test suite, for example the multimaster tests, again, it required an update to runtests.py and quite a bit of code duplication.

There needed to be a better way going forward.

What's The Gain?

If you know PyTest, you already know what it brings to the table. If you're not familiar, here are a few highlights:

If you need "something" added to your test suite, it's likely that someone already created a plugin implementing that.
Selecting a group of tests, a single test, a test class, is incredibly poweful and simple, especially when compared to runtests.py.
- Running all fileserver related tests:
```
nox -e 'pytest-zeromq-3(coverage=False)' -- -k fileserver
```
- Running all proxy related tests:
```
nox -e 'pytest-zeromq-3(coverage=False)' -- -k proxy
```
- Running windows specific tests(which commonly contain win in either the test module name or test name):
```
nox -e 'pytest-zeromq-3(coverage=False)' -- -k win
```
- Running the failed tests after a test run, stopping at first failure:
```
nox -e 'pytest-zeromq-3(coverage=False)' -- --lf --maxfail=1
```
Fixtures!

These are powerful and bring code deduplication to the table. The less code you have to maintain, the better, especially when talking about the test suite. Imagine you have a big setup routine to prepare for a test. In the unittest world you would use setUp or setUpClass. Now, what if that setup routine had to be used in multiple test classes? Likely you would either duplicate code or create a base class to reuse the code. In PyTest, you create a function, the fixture, and every test that needs that prep work just needs to add the function name(fixture name) as an argument to the test function, and

it "Just Works!"

You can even parametrize and provide scope for the fixtures!

Assertions!

PyTest is smart enough that all you need to do on your test function is assert something. It knows how to handle python's builtin types and more if needed. Also, in case of failure, it adds useful information on the error output to hint as to why it failed. For example:

$ pytest test_assert2.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-6.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_assert2.py F                                                    [100%]

================================= FAILURES =================================
___________________________ test_set_comparison ____________________________

    def test_set_comparison():
        set1 = set("1308")
        set2 = set("8035")
>       assert set1 == set2
E       AssertionError: assert {'0', '1', '3', '8'} == {'0', '3', '5', '8'}
E         Extra items in the left set:
E         '1'
E         Extra items in the right set:
E         '5'
E         Use -v to get the full diff

test_assert2.py:6: AssertionError
========================= short test summary info ==========================
FAILED test_assert2.py::test_set_comparison - AssertionError: assert {'0'...
============================ 1 failed in 0.12s =============================

There's a lot more to gain by using PyTest and also a lot of resources on the internet that you can read to know more.