.. _tox:
Running Commands with Tox
=========================
`Tox `__ is a general purpose tool for
automating Python testing. We recommend using tox to specify the environments
in which your tests are run, both locally and on :ref:`ci` services.
Getting Started with Tox: Running tests
---------------------------------------
The first thing to configure tox to do is to run the tests for a package. The
most minimal tox file for a package following this guide is:
.. code-block:: ini
[tox]
envlist = py38
isolated_build = True
[testenv]
extras = test
commands = pytest {posargs}
Let's dig into the sections of this file, the ``[tox]`` section is the `global
configuration
`__ for
the whole file. We use this to define ``envlist`` which is a list of all the
different builds configured in tox, here we set this to be a Python 3.8
environment, we will expand on this shortly. The ``isolated_build``
configuration option configures tox to build your source distribution in the
same manner as recommended in :ref:`releasing`.
The ``[testenv]`` section describes settings common to all environments you
specify in the tox file (unless they are later overridden), here we default
the ``commands =`` option to run pytest. The ``{posargs}`` is a tox
`substitution
`__ which
passes extra arguments through to ``pytest``.
The ``extras = test`` line tells tox to install the ``optional-dependencies`` section
listed in ``pyproject.toml`` for running your test suite; this should include ``pytest``.
To run your tests with tox run:
.. code-block:: console
$ tox -e py38
To pass arguments through to ``pytest`` use ``--`` here we tell pytest to
stop after the first failure.
.. code-block:: console
$ tox -e py38 -- -x
Multiple builds
###############
Tox allows configuration of multiple builds in a few different ways, the
easiest one is to specify multiple Python versions in the env list:
.. code-block:: ini
[tox]
envlist = py{37,38}
isolated_build = True
This takes our one test configuration and makes a Python 3.7 and a Python 3.8
environment that can be seen by listing all tox environments with:
.. code-block:: console
$ tox -l
py38
py37
This feature is called `generative envlist `__ and can be used to create many build environments with minimal repetition.
Named Environments
##################
Using generative build environments you can define extra named environments
which can be useful for builds that need to specify specific dependencies or
settings. So far on this page we have assumed that all your dependencies are
specified in :ref:`pyproject`. You can extend or override this by using the
``deps =`` configuration option in tox. Here we define a named test
environment which installs the development version of numpy.
.. code-block:: ini
[tox]
envlist = py{37,38}{-numpydev,}
isolated_build = True
[testenv]
extras = test
commands = pytest {posargs}
deps =
numpydev: git+https://github.com/numpy/numpy
the ``envlist`` is now more complex, the result of this the following:
.. code-block:: console
$ tox -l
py37-numpydev
py37
py38-numpydev
py38
with the ``deps`` overridden for ``numpydev`` builds.
Environment variables
#####################
It is often useful to set environment variables within the building and testing
environment prior to testing. Environment variables can be set within ``tox.ini``
with:
.. code-block:: ini
[testenv]
# Pass through the following environment variables which may be needed for the CI
passenv = HOME, WINDIR, LC_ALL, LC_CTYPE, CC, CI, TRAVIS
# Suppress display of matplotlib plots generated during docs build
setenv = MPLBACKEND=agg
The variables listed after ``passenv`` will be preserved from the
environment that you used to run tox, while the ``setenv`` variables
are set within the testing environment. In the template, we have set the
``MPLBACKEND`` variable to the ``agg`` backend, which prevents matplotlib
from launching interactive plot displays when generating figures from the
matplotlib plot directive or pytest-mpl. For more on making use of this
feature, see :ref:`plot_directive`.
Building Documentation with tox
-------------------------------
One common task which isn't running the test suite is building sphinx
documentation, documentation builds can be complex with a number of extra
dependencies or settings. In this section we will add a ``build_docs`` named
environment to tox. This section assumes you have already followed
:ref:`documentation`.
.. code-block:: ini
[testenv:build_docs]
extras = docs
commands = sphinx-build docs docs/_build/html -W -b html {posargs}
This section installs the package extras for the documentation, which should
be a list of all your documentation dependencies and then sets the command to
be the `sphinx-build
`__ command to
build the docs and output them in the ``docs/_build/html`` folder relative to
the ``tox.ini`` file.
You can now run your documentation with:
.. code-block:: console
$ tox -e build_docs
you can pass through extra arguments to `sphinx-build
`__ because of
the ``{posargs}`` substitution. For example to force sphinx to ignore its
cache you can run:
.. code-block:: console
$ tox -e build_docs -- -aE
Testing Packages with Compiled Extensions
-----------------------------------------
As configured in this guide so far, tox will perform the following actions (all in the same directory as the ``tox.ini`` file):
1. ``python setup.py sdist``
2. Create a new virtualenv
3. Install the built sdist.
4. Run the commands listed in ``commands =``, which here we assume to be ``pytest``.
(See https://tox.readthedocs.io/en/latest/index.html#system-overview for more details.)
For packages laid out as described in this guide, i.e. with the Python
package in a directory in the root repo, i.e. ``astropy/``, this means that
when ``pytest`` is run, it will collect the tests from the local directory
(as desired), and all imports of the package i.e. ``astropy`` will be
imported from the local directory *not the installed sdist*.
For pure python packages this generally isn't a problem, the contents of the
installed sdist and the local directory are the same (tox just made the sdist
from the local directory). However, for packages that include compiled
extensions, the installed package and the local directory are *not the same*.
The installed package has build the compiled extensions, and the local
directory does not. This means that unless you make some adjustments to the
package or the tox configuration compiled extensions will not work when
running pytest through tox as described above.
There are two main ways to alleviate this issue:
1. Move the Python package source code under a ``src/`` folder in the root of
the repo. This is a common package layout for Python projects, and it means
that you can not import your package relative to the git root, meaning it
will be imported from the installed sdist, see https://setuptools.readthedocs.io/en/latest/setuptools.html#using-a-src-layout for details.
2. Configure tox to run ``pytest`` from a temporary directory so that the
local import does not work. With this method you make use of pytest's
`--pyargs flag
`__
to run the tests against the installed version of the package. This ensures
that any compiled extensions are properly detected, but prevents things like
specifying paths to pytest from working.
To configure tox to run ``pytest`` from a temporary directory do the
following in ``tox.ini``:
.. code-block:: ini
[tox]
envlist = py38
isolated_build = True
[testenv]
changedir = tmp
extras = test
commands = pytest --pyargs packagename {posargs}
replacing ``packagename`` with the name of your package as you import it,
i.e. ``astropy``.