Testing Python projects

Use Tox to run tests across multiple versions of Python. Tox installs the package into a virtualenv for each environment and runs the test command inside that.

Most projects should use pytest, a testing runner and framework that is much easier to use that the unittest module.

A basic tox.ini file to use pytest should look like this:

[tox]
minversion=1.8.0
envlist=py{26,27,33,34},lint,docs

[testenv]
commands=py.test {posargs} <MODULE>
deps=pytest

[pytest]
addopts=-q

Configuring Travis-CI to run Tox properly is slightly verbose, but means local and remote tests run in an almost identical fashion. The .travis.yml file should include each tox environment in it’s matrix (see the Configuration section below for a full example):

language: python
env:
  - TOXENV=py26
  - TOXENV=py27
  - TOXENV=py33
  - TOXENV=py34
  - TOXENV=lint
  - TOXENV=docs
install:
  - pip install tox
script:
  - tox

Style and lint checking

Use flake8 to run style and lint checks, which collects pyflakes, pep8 and mccabe. The flake8-docstrings module extends it to include pep257. As it reads it’s configuration from tox.ini, it can be added to Tox very easily.

[testenv:lint]
commands=flake8 <MODULE>
basepython=python2.7
deps=
    flake8
    flake8_docstrings

[flake8]
ignore=D102,D203

This ignores the following messages:

D102: Function docstring missing
D203: Expected 1 blank line *before* class docstring, found 0

Configuration

A full Tox configuration file for a standard python package should run tests on each supported version of Python, run style checkers, and include environments to build documentation and release the package (see the Python Packaging page). This configuration allows packages to be built and tested without needing to install any dependancies other than Tox.

[tox]
minversion=1.8.0
envlist=py{26,27,33,34},lint,docs

[testenv]
commands=py.test {posargs} <MODULE>
deps=pytest

[pytest]
addopts=-q

[testenv:lint]
commands=flake8 <MODULE>
basepython=python2.7
deps=
    flake8
    flake8_docstrings

[flake8]
ignore=D102,D203

[testenv:docs]
commands=sphinx-build -qE docs/ docs/_build/
deps=
    sphinx
    sphinx_rtd_theme

[testenv:release]
commands=python setup.py sdist bdist_wheel upload
deps=wheel

Travis-CI should be configured to run all environments (excluding releasing the package), though style and documentation environments should be allowed to fail. Including sudo: false will use Travis-CI’s container based build system.

language: python
env:
  - TOXENV=py26
  - TOXENV=py27
  - TOXENV=py33
  - TOXENV=py34
  - TOXENV=lint
  - TOXENV=docs
install:
  - pip install tox
script:
  - tox
matrix:
  allow_failures:
    - env: TOXENV=lint
    - env: TOXENV=docs
sudo: false