XClose

Information Services Division

Home
Menu

Automated testing for research

Overview

An important step to achieving reliability in a research software project is to develop a suite of tests alongside the software which can be run at regular intervals to ensure that it builds, runs, and produces correct and consistent results.

To ensure that tests are always passing, we recommend use of a continuous integration (CI) server. The CI server is configured to build the code and execute tests and report any failures. CI servers can also be configured to test pull requests on Github and report status back directly in the Github interface.

For this purpose RSDG recommends the use of Travis-CI. Travis-CI is a cloud-based service that runs tests on Linux and OSX. Travis-CI exists in two different versions for public and private repositories.

Travis-CI.org is free for use on public repositories. We encourage and will support all UCL researchers with code in public github repositories to make use of this service.

Travis-CI.com is a non-free service suitable for private Github repositories. UCL will cover the cost of this service to researchers at UCL. All private repositories hosted within the UCL github organisation now have automatic free access to use Travis-CI.com. If you wish to use the service on a repository hosted in a different github organisation please get in touch via rc-softdev@ucl.ac.uk

User guides and terms of service

For more detailed advice on how to use the Travis-CI service please see:

For terms of service see:

Alternatives

For some research code bases Travis may be unsuitable. Typical examples where Travis-CI is unsuitable are:

  • Highly parallel code-bases i.e. codes making use of MPI, OpenMP or similar designed to run across multiple cores/nodes. Travis is unsuitable as worker nodes have a maximum of 2 CPU cores.
  • Code that relies on paid-for dependencies such as Matlab or Intel compilers.

For such projects we recommend use of the STFC SESC Build Service.