UKCOMES: UK Consortium on Mesoscale Engineering Sciences

1. To advance the emerging mesoscale science and engineering through exploitation of national high-end computing (HEC) resources.
2. To bring together all the required expertise to make critical theoretical discoveries and model developments and translate these into software that is able to exploit today's and future HEC hardware.
3. To upgrade the DL_MESO suite into a world-leading open-source software infrastructure for mesocale and multiscale modelling and simulation that is readily accessible to both academy and industry.
4. To enable cutting-edge simulations in strategically important areas on national HEC platform to generate high impact research output.
5. To connect the UK and global research and the end-user communities, e.g. in industries, in order to develop mesoscale sciences that underpin vital technologies for UK industry and economy.
6. To provide a stimulating, collaborative and interdisciplinary environment to train future generations in the relevant fields.

UKCOMES's main HEC platform is the UK National Supercomputing Service, ARCHER, which is currently ranked No. 25 among the Top 500 Supercomputers in the world. Moreover, it is one of the very top supercomputers in the world dedicated to academic research. It is based on the Cray XC30 architecture, Intel Xeon E5 v2 12C 2.700GHz processors, and Aries interconnect. It has 133,824 cores in total and a peak speed of 2.569 Petaflops.

Dr Rupert Nash of EPCC serves as the ARCHER contact for UKCOMES.

In addition, consortium members have access to a range of local supercomputers. For example, the Blue Joule at Daresbury Laboratory - STFC has a total of 131072 processors and a peak speed of 1.678 Petaflop, which is ranked No. 30 among the Top 500 Supercomputers in the world.

• Prof Kai Luo (UCL, PI)
• Dr Santosh Ansumali (Jawaharlal Nehru Centre for Advanced Scientific Research, India)
• Dr Miguel Bernabeu (The University of Edinburgh)
• Dr Edo Boek (Queen Mary, University of London)
• Dr Sheng Chen (University of Nottingham)
• Prof Peter Coveney (UCL)
• Dr Paul Dellar (University of Oxford)
• Prof Lyazid Djenidi (University of Newcastle, Australia)
• Dr David Emerson (Daresbury Laboratory, STFC)
• Prof Claus Feuchter (University of Aalen, Germany)
• Dr. Tiziano Ghisu (University of Cagliari, Italy)
• Dr David Graham (Plymouth University)
• Dr Derek Groen (Brunel University) Dr David Holman (Next Limit Technologies, Spain)
• Dr Ian Halliday (Sheffield Hallam University)
• Dr Timoleon Kipouros (University of Cambridge)
• Dr Timos Kipouros (Cranfield University)
• Dr Timm Krüger (University of Edinburgh)
• Dr Halim Kusumaatmaja (Durham University)
• Dr Krzysztof Kubiak (University of Huddersfield)
• Prof Kurt Langfeld (Plymouth University)
• Dr Rodrigo Ledesma-Aguilar (Northumbria University)
• Prof Qing Li (Central South University, China)
• Dr Daniel Lycett-Brown (Dassault Systemes Simulia, Spain)
• Dr. Jianping Meng (Daresbury Laboratory, STFC)
• Dr Rupert Nash (EPCC, consortium ARCHER contact)
• Prof Tim Phillips (Cardiff University)
• Prof Rongshan Qin (Open University)
• Dr Tim Reis (University of Greenwich)
• Dr Alistair Revell (University of Manchester)
• Dr. Daniel Ruprecht (Univeristy of Leeds)
• Prof Mark Savill (Cranfield University, Chair of CCPEngSci)
• Dr Michael Seaton (Daresbury Laboratory)
• Dr Ulf Schiller (Clemson University, USA)
• Dr Jon Summers (University of Leeds) 
• Dr Sebastian Schmieschek (UCL)
• Dr Ilian Todorov (Daresbury Laboratory, STFC)
• Dr Mark Wilson (University of Leeds)
• Prof Yuying Yan (University of Nottingham)
• Prof Julia Yeomans FRS (University of Oxford)
• Prof Yonghao Zhang (University of Strathclyde) 

• Prof Kai Luo (Chair of MC, UCL)
• Dr Miguel Bernabeu (The University of Edinburgh)
• Prof Peter Coveney (UCL)
• Prof David Emerson (Daresbury Laboratory, STFC)
• Prof Tim Phillips (Cardiff University)
• Prof Rongshan Qin (Open University)
• Prof Mark Savill (Cranfield University)
• Prof Yonghao Zhang (University of Strathclyde) 

• Prof Sauro Succi (Italian Institute of Technology Foundation and University of Harvard, Chair of SAB)
• Prof Ilya Karlin (ETH Zurich)
• Prof Lyazid Djenidi (University of Newcastle, Australia)
• Prof Andrew Pollard (Queen's University, Canada)
• Prof Xiaowen Shan (Southern University of Science and Technology, China)
• Prof Stefan Turek (Technical University Dortmund, Germany)
• Prof Wen-An Yong (Tsinghua University, China)

• Dr Patrick Warren (Unilever R & D, Chair of IAB)
• Dr Ir. Jan Willem van der Burg (Airbus)
• Dr David Holman (Dassault Systèmes Simulia)
• Dr Duan Wu (Mitsubishi Electric R&D Centre Europe BV)
• Prof Guangwen Yang (National Supercomputing Center in Wuxi)

Lead: Michael Seaton

This work package is involved with developing efficient and highly scalable code for a variety of hardware platforms to perform mesoscale modelling, primarily using the Lattice Boltzmann (LB) method, for science and engineering problems.

The LB code contained in the general-purpose mesoscopic simulation package DL_MESO (as originally developed for CCP5) is seen as the main production code for UKCOMES and designed to work on CPU-based systems with MPI. Development work on this code is under way to incorporate new functionalities required for other work packages and to disseminate them across the LB community in the UK and further afield.

Complementing DL_MESO, two codes developed at the Centre for Computational Science at UCL are currently maintained by the consortium- HemeLB and LB3D. Optimised for the sparse geometries of blood vessel networks, HemeLB is developed with assured accuracy and user friendliness in mind. The long term vision for HemeLB includes its use in clinical practice. LB3D is a feature rich MPI parallel FORTRAN code for the simulation of complex fluids. While the functionality of LB3D has some overlap with DL_MESO, its user base and functionality justify parallel development. Here efforts are coordinated to create synergies rather than redundancies.

Optimisation and maintenance of the LB code in DL_MESO are essential to ensure it can continue to run on high-end computing platforms as required for solving challenging scientific and engineering problems at the mesoscale. Multithreading with OpenMP, algorithmic and compile-time improvements are being implemented to improve the code's efficiency and parallel scalability. Further optimisations for running on coprocessor (e.g. Intel Xeon Phi) and GPU-based systems using programming platforms such as CUDA are also being considered, guided by recent developments within work package participants' own research codes, to exploit the increases in performance available from such architectures. Code maintenance is dependent on the needs of users: beyond functionality development and optimisation, this includes increasing the numbers of users, improving ease-of-use and providing user support and training. 

Lead: Yonghao Zhang

 Some examples of micro and nano flows are:

1. Gas flows: We will apply high-order Lattice Boltzmann models for gas flows in micro and nano devices, aiming to establish practical design simulation tools to enable next generation technology such as design of future lithographic machines, and to improve our understanding of gas transport mechanisms through ultra-tight porous media.
2. Microfluidic microdroplet technologies: A new design simulator will be developed for designing and optimising droplet-based microfluidic systems, in place of the currently dominant empirical approaches, leading to a technological leap in robust, highly automated and integrated droplet-based microfluidic technologies.
3. Microdroplet collisions under various conditions: So far, the majority of the experimentally observed collision regimes have been reproduced by the Lattice Boltzmann Method. Further model development and large scale simulations will hopefully cover all regimes, with simulated physical parameters approaching those in the experiments.
4. Multiphase flows in porous media.

Lead: Rongshan Qin

The work package thermal application is concerned with the development and application of mesoscopic simulation methods for understanding thermal fluids

Lattice Boltzmann Methods for Thermal Flow
Lattice Boltzmann methods are continually being developed for thermal flow to cover the full range of temperatures and temperature variations. Single-distribution and double-distribution function LBM as well as hybrid LBM-finite difference methods are considered.

Coupling hydrodynamics with engineering thermodynamics
The mesoscopic interparticle potentials are derived from the engineering thermodynamics and then substituted into the mesoscopic hydrodynamic models. This enables to investigate the interactions between fluid flow and phase transformation.

Materials processing simulation
Current efforts include the simulation of blast furnace processing, super-clean steelmaking and fast solidification. Some external fields, e.g. pulsed electric field or electropulsing, are implemented in the investigations. 

Particle-based mesoscopic methods
Smoothed particle hydrodynamics and dissipative particle dynamics code packages are developed to address the engineering thermal fluid problems. Multiphase and multicomponent systems are the primary interests in this work package.

Lead: Tim Phillips

Lattice Boltzmann Methods for Multiphase Flow
Lattice Boltzmann methods are continually being developed for multiphase flow to cover the full range of physical parameters (such as the density ratio, Reynolds number, Weber number, capillary number, Ohnesorge number). Recent efforts have been focused on improving Shan-Chen pseudopotential models and obtaining high-order models such as the cascaded LBM. Multicomponent LBMs are also being developed.

Lattice Boltzmann Methods for Non-Newtonian and Viscoelastic Fluids
Most existing macroscopic models for viscoelastic fluids fail at high fluid elasticity. Whereas models based on kinetic theory are much more robust, they are computationally expensive due to the size of configuration space. Work in the UKCOMES is focused on developing efficient and accurate LBMs for non-Newtonian and viscoelastic fluids in which conventional LBM treatment of Newtonian fluids is coupled to a treatment of the rheological equation of state. A considerable amount of experimental data has been used to benchmark LBM predictions for viscoelastic flow problems such as flow past a cylinder or sphere and flow through contraction geometries

Complex Fluids simulation
Current activities include (i) predictive modelling of dynamic wetting phenomena; (ii) industrial inkjet printing, where colloidal suspensions are used for additive manufacture; (iii) separation of water from diesel (especially biodiesel, where surfactants lead to low interfacial tensions) via fuel filtration systems; (iv) enhanced oil recovery, where alteration of rock wettability and direct simulation of multiphase flow through actual pore structures is vital; (v) nanofluid modeling for enhanced cooling applications; and (vi) computational tribology, where the aim is to develop predictive models of tribofilm formation (and surface wear) to enable development of environmentally friendly lubrication systems for internal combustions engines to reduce emissions and improve efficiency; and (vii) flow control using pulsed electric fields.

Lead: Kai Luo

Lattice Boltzmann Methods for High Speed and Turbulent Flow
Lattice Boltzmann methods have been traditionally used for low Reynolds number flow. Recent developments have extended the LBM to high speed and high Mach number flow as well as turbulent flow. One approach is to develop various high-order LBMs by extending their stability limit. The other is to incorporate "sub-grid scale" models, essentially conducting a large eddy simulation.

Lattice Boltzmann Methods for Reactive Flow
Being a kinetic method, LBM is well suited to simulation of multi-species and/or reactive flow with chemical reactions. A number of techniques have been developed to couple the flow field and the temperature field. One approach is the double distribution model in which the temperature (or enthalpy) field is represented by a separate distribution function within the LBM framework. Another approach is to adopt a hybrid method in which the flow field is determined by LBM while the energy equation is solved by a continuum method such as the finite difference. Multi-step chemistry can also be incorporated in the simulation.

High-end Computing and Real-time Simulation
With its intrinsically high parallel efficiency, LBMs are well placed to exploit the rapidly advancing high-end computing. Real-time simulation refers to the situation where the simulation time on computer is equal to or shorter than the physical time of a physical process so that we can follow or alter a physical process instantaneously. Work in the UKCOMES has combined LBM simulation on GPUs with fast 3D rendering engine to monitor and control   complex flow in realistic environments. Potential applications range from fire fighting to pathogen control in hospitals.

Lead: Peter Coveney

The work package multiphysics phenomena is concerned with the development, application and optimisation of hybrid simulation methods for high performance computing applications in biomedicine and engineering. 

Coupling across scales
At the centre of efficient modelling of multiphysics phenomena lies the coupling of algorithms in general and specific codes in particular. Efforts in this domain range from the very small to the very large scales including molecular systems as well as Earth System Modelling.

Explicit particle models for blood flow simulation
Current efforts include the integration of the lattice-Boltzmann implementation HemeLB with a fully resolved red blood cell model based on the immersed boundary formalism. As well in HemeLB, a coarse grained friction coupled point-like colloid model is extended to include magnetic interactions towards the modelling of magnetic drug targeting utilising ferro-fluids. 

Wetting effects involving multiple fluid components and anisotropic particles
The multicomponent lattice-Boltzmann application LB3D allow to include (anisotropic) particles with a wettabilty model. Large-scale simulations of systems with wetting anisotropic particles oriented by external fields have revealed self-organisation effects by hydrodynamic dipole-dipole interaction of menisci.

Incremental coarsening of MD simulations in material science
Integrating CPMD ab-initio simulations and full-atom and coarse grained MD simulations by Amber overall model detail can be reduced while fully resolving properties in critical areas. In context of research into the swelling behaviour of clays this allowed to take several layers of clay material into consideration while at the same time preserving the crucial potential make-up at the layer edges.

Lead: Mark Savill

The use of LBM with LES, as considered by Savill's group at Cranfield, is one of several ways in which LBM can be incorporated in an appropriate hierarchy of CFD methods to address multi-scale fluid effects. This is required if we are to move beyond current established multi-objective, multi-disciplinary, and multi-physics design optimisation capabilities to address true multi-scale simulation and optimisation; an objective which has been recognised as one of the current Engineering Grand Challenges identified by EPSRC. The fact that LBM can handle multi-phase flow with phase change opens up the possibility of extending such optimisation to fluid-solidification problems in a more natural manner than has previously prove possible. In particular existing work on aerodynamic icing using coupled Navier-Stokes and Lagrangian water droplet tracking codes, with specialist splashing, wetting, and thin -film heat transfer treatments, may be superseded by use of LBM for both the core and surface flow interactions. This is already the subject of work at Cranfield with recent Airbus/EPSRC CASE PhD & EngD activities continuing via ATI and MSc projects. 

Alistair Revell (Manchester University) is separately pursuing the use of LBM in conjunction with Smooth Particle Hydrodynamics as Co-Investigator for an EPSRC X-Med project on Extreme Loading of Marine Energy Devices due to waves, currents, flotsam and mammal impact. This provides an opportunity to compare and contrast these two mesh-less solution methods for impact of rigid and flexible objects with a moving turbine, and thus enable their cross-validation. Such model validation remains crucial to build confidence in handling such challenging and highly complex real-world fluid-structure interactions.  

A first WPG workshop was held at UCL on December 17th, 2014, attended by 10 representatives of participating consortium groups with presentations by Cranfield (Prof.Savill & Dr. Timos Kipouros); UCL (Prof. Kai Luo & Dr. Daniel Lycett-Brown; Leeds (Dr. Mark Wilson & Nicholas Delbosc); and attendance of Michael Seaton (Daresbury STFC), Rupert Nash (Edinburgh EPCC), Joe O'Connor (Manchester for Revell).  A second WPG discussion group meeting , with inputs from groups at Cambridge (Kipouros), Cranfield (Savill), Huddersfield (Krysztof Kubiak), Leeds (Summers) and Sheffield (Rongshan Qin), was held as a session within the UKCOMES Annual Workshop at UCL on December 17th , 2015; with representatives from Brunel, Daresbury, EPCC, Manchester, NextLimit Dynamics, and UCL also in attendance.  These workshops identified the exciting possibilities opening up for topology optimisation and real-time simulation/optimisation, as well as new prospects for deploying LBM for acoustic analyses.

The work of WPG within UKCOMES has also been reported and discussed at successive STFC CCP Steering Panel meetings at Daresbury and Rutherford Appleton Laboratories.  A wider collaboration across the work package topics, involving consortium participants as well as co-operating groups outside the UK, has been initiated also with additional Early Access Support from the STFC Hartree Centre for comparative NS and LBM Simulations by separately funded RA and PhD researchers.

Two (international and national) test cases have been established: a baffled micro-reactor configuration for fluid mixing; and a micro-combustor geometry for electrical power generation or thrusters propulsion - URANS, LES and DNS data are already available for these to allow multiscale evaluation against LBM. 

 Additionally Krysztof Kubiak at Huddersfield University has now extended the Leeds multi-phase LBM research code to design optimisation for super hydrophobic surfaces, while other groups have started using LBM for simpler parametric optimisation applications - in particular non-obtrusive electric field methods for controlling particulate impurities in molten metal casting  by Rongshan Quin at Sheffield University.

CPU-based work initiated with the first consortium PDRA (Daniel Lycett-Brown) for cross-comparing the Southampton/UCL LBM code with the Cranfield one in 2D and then 3D for the micro-reactor case, will be completed once a further researcher is in place;  before moving onto DL_MESO for the micro-combustor mixing case. This will then allow the extension to combustion LBM to be explored.

Daniel Lycett-Brown has meantime moved to NextLimit Dynamics who promote the commercial LBM code XFlow; joining Professor Savill's former EngD student, Giuseppe Trapani, and being joined by Nicholas Delbosc in 2016. This has opened up the possibility to greatly strengthen our industrial collaboration, with a focus especially towards aeroacoustic/MDO applications and more effective interactive optimisation generally. 

A  first integration of LBM for micro-fluidic design optimisation within a GPU environment has been achieved at Cranfield; speeding up the process by a factor of more than 50, resulting in a much richer and more detailed design space exploration.  Further extension to incorporate a more flexible, level-set geometry representation has now provided even further speed up and allowed very general parameterisation required to move towards fully topological Multi-Disciplinary Optimisation. A next development will involve coupling and sharing of code between the Cambridge/Cranfield and Leeds groups to incorporate on-the-fly visualisation within a fully interactive design optimisation environment.

The experience gained in porting under CUDA has already benefitted GPU porting by Daresbury of the UCL medical flow code HEME_LB for patient-specific computational analysis, and ultimately guided surgical intervention. Similar attention will next be transferred to porting DL_MESO also.