Automated testing for research
A good approach to achieving reliability in a research software project is to develop a suite of tests alongside the software which can be run at regular intervals to ensure that it builds, runs, and produces correct and consistent results.
A good way to ensure that tests are always passing is to use a continuous integration (CI) server. The CI server is configured to build the code and execute tests and report any failures. CI servers can also be configured to test pull requests on Github and report status back directly in the Github interface.
For this purpose RITS is recommending the use of Travis-CI. Travis-CI is a cloud-based service that runs tests on Linux and OSX. Travis-CI exists in two different versions for public and private repositories.
Travis-CI.org is free for use on public repositories. We encourage and will support all UCL researchers with code in public github repositories to make use of this service.
Travis-CI.com is a non-free service suitable for private Github repositories. UCL will cover the cost of this service to researchers at UCL. All private repositories hosted within the UCL github organisation now have automatic free access to use Travis-CI.com. If you wish to use the service on a repository hosted in a different github organisation please get in touch via email@example.com
User guides and terms of service
For more detailed advice on how to use the Travis-CI service please see:
For terms of service see:
For some research code bases Travis may be unsuitable. Typical examples where Travis-CI is unsuitable are:
- Highly parallel code-bases i.e. codes making use of MPI, OpenMP or similar designed to run across multiple cores/nodes. Travis is unsuitable as worker nodes have a maximum of 2 CPU cores.
- Code that relies on paid-for dependencies such as Matlab or Intel compilers.
To assist researchers with codes unsuitable for Travis-CI we have developed an additional solution based on Jenkins, which allows researchers to run tests on a subset of the UCL supercomputer Legion. This setup includes the full Legion and Grace software stack and modules system, so is ideal for those developing HPC codes to run on these platforms.
We are actively seeking users for these services - please email firstname.lastname@example.org if you are interested in helping and have a working suite of unit or regression tests for your code, or would like help in developing one.