Optimised tidal modelling
Posted: 2 Feb 2017 | 11:37
Fluidity for tidal modelling
Figure 1: Mesh for the Sound of Islay tidal simulation. Courtesy Dr Creech.
We were recently involved in a project to optimise the CFD modelling package Fluidity for tidal modelling. This ARCHER eCSE project was primarily carried out by Dr Angus Creech from the Institute of Energy Systems in Edinburgh.
Fluidity is a highly capable modelling package, but as demonstrated in Figure 2, does not scale especially well for the tidal models we are interested in performing (an example is shown in Figure 1). Angus identified a number of areas where performance optimisation was possible:
Figure 2: Weak scaling performance of the unoptimised code on ARCHER. Courtesy Dr Creech.
- Serial optimisation of the construction of the discontinuous galerkin (DG) matrix.
- Mesh re-ordering.
- Load balancing.
The routines that construct the DG matrix used in the turbulent flow calculations are generalised to allow many different types of simulation. However, this has an impact on the performance as it inhibits the compilers' ability to optimise these key computational routines.
To optimise these routines for tidal simulations, more specialised versions were created that converted dynamic array allocations to static allocations, added compile time loop length definitions to enable the compiler to vectorise the loops, and inline the main computational routines.
This was combined with some code that selects these optimised routines at run time if the correct type of simulation is being run, and falls back to the original code for functionality that hasn't been optimised.
For the type of simulation we are optimising for in this work, Fluidity extrudes a 2D mesh into a 3D mesh for the calculations. The finite element assembly code then works through this 3D mesh, considering elements one at a time and using data from neighbouring elements. However, the way the 3D mesh is generated does not take into consideration the type of elements created, and the requirement for neighbouring element data. This leads to meshes that are not well ordered for the calculations being undertaken.
With the help of some code from Dr James R. Maddison the 3D mesh was re-ordered to take into account element types, and thus optimise the element-by-element operations.
For the simulations we are optimising for, the existing mesh decomposition/redecomposition tool creates meshes with considerable imbalance in their relative size. Some straightforward changes were implemented to take into consideration the number of nodes below each surface when decomposing the mesh, and adding that value to the weight of a mesh partition. This allows the current mesh partitioning functionality to create a much more balanced decomposition, moving from around a 6x load imbalance (at maximum) to a 2x load imbalance (at maximum) for a representative tidal simulation.
With these three optimisations implemented, we were able to achieve a speedup of over 3x for a representative simulation on 768 cores on ARCHER (as shown in Figure 3), representing a very significant saving in terms of computational resources and enabling new scientific research to be undertaken.
Figure 3: Performance improvement for the different optimisations on 768 cores on ARCHER. Image courtesy Dr Creech.
We also developed a new large eddy simulation (LES) turbulence model, in which continuous galerkin (CG) velocity and eddy viscosity fields were used rather than DG fields. This implementation gave a more efficient and simpler formulation than would have been possible using a solely DG approach.
Using all the functionality implemented in this project, Angus was able to run highly detailed tidal simulations of the Sound of Islay and demonstrate excellent agreement between the simulated data and satellite images of the actual tidal flow at the site, as demonstrated in the picture below. This means that Fluidity can be efficiently used for tidal energy simulations with a high degree of confidence in the accuracy of such simulation.
If you're interested in using this modelling functionality on ARCHER, or have any questions/comments about this work please don't hesitate to get in touch.