DiRAC: supporting the UK's particle physicists and astrophysicists

DiRAC (Distributed Research Utilising Advanced Computing) provides distributed high performance computing (HPC) services to the Science and Technology Facilities Council (STFC) theory community. In practice this means providing particle physicists and astrophysicists with both data and compute resources to run their simulations and process the data produced. 

EPCC is involved in DiRAC in several ways. It is one of the sites providing HPC facilities - the others being Cambridge, Leicester and Durham. At EPCC I am the Technical Manager of the RAC (Resource Allocation Committee). The RAC is the committee which decides how much resource each research group gets, and the systems on which each project is hosted.

EPCC also runs the SAFE, which is the online service management system used to manage allocations, create accounts, report usage, and is where the helpdesk is run. 

The system we host is the GPU-based Tursa system, which came on-line in 2021 and was upgraded in 2023. Tursa presently consists of 114 GPU nodes, each with four NVIDIA A100-40 Tensor Core GPUs and 64 of the higher memory A100-80 nodes, which were added in 2023. Nodes are connected with a very fast Infiniband Interconnect, allowing large calculations to run efficiently on many GPU at once.

Previously Edinburgh hosted Tesseract, which was decommissioned in 2023 after around five years of running and providing DiRAC with extreme scaling CPU-based hardware.

Each of the DiRAC systems has particular features (e.g. data or memory intensive, extreme scaling, CPU-based or GPU-based, etc) which make each of them appropriate for different kinds of research. This is where the RAC comes in…

Resource Allocation Committee (RAC)

The main part of my role is to manage resource allocations. Across the HPC community generally the frequency of issuing calls varies. For DiRAC the RAC calls are issued yearly, with occasional extra calls if resources become available mid-year. As this is a yearly process, and projects can run for up to three years, this is a large exercise which is usually started in the spring when we assess what resources are available. Calls then open in the summer and close in the autumn, with the winter months used for reviewing. 

First, technical reviews are carried out (this is something I organise within the technical manager role). The proposals and completed technical reviews are then passed to the research council involved (STFC) who complete the scientific reviews. Then, once all of the reviews are complete, the allocation of resources is carried out, with projects finally starting in April.

As well as overseeing the technical side of the RAC process, the role requires a few other things. For example researchers who need quick access to resources to test out their code can apply for a seedcorn project which will have to be set up. Likewise occasionally we have discretionary awards to organise. Then of course there are the inevitable tasks such as reporting on allocations, usage, and much more! 

To see how Tursa supports the STFC theory community, see the examples below.

DiRAC in action

Two projects awarded resources on Tursa are described here. 

Modelling misaligned accretion discs around supermassive black holes (SMBHs) 
Joshua Stanway, University of Central Lancashire
jsstanway@uclan.ac.uk

See images at top of page and above.

Images from the Event Horizon Telescope (EHT) uncover the first look at a black hole’s accretion disc just outside the event horizon. This previously unexplorable region is now a laboratory for testing accretion physics on new scales. One key area of theoretical development is relaxing and reducing previously made assumptions in modelling black hole systems, which is what I aim to achieve throughout this RAC project. My group's research focuses on misaligned accretion disc models, where the accretion disc does not start perpendicular to the spin axis of the black hole, as is common in previous work. This can lead to significant differences in the structure of the accretion disc, where at large radii the disc remains tilted, but due to frame dragging around the black hole, the inner edge is forced to align. We see this effect in the top row of the figure. The bottom row of the figure shows snapshot images of a misaligned disc, first the total intensity, and then linear and circular polarisation. 

With the misaligned simulations produced throughout this RAC project, we will produce one of the world's largest misaligned disc image libraries to compare the EHT observations, to learn more about how an initially misaligned disc differs from “standard” models of black holes.

Thermodynamics and MHD of the inner regions of protoplanetary discs 
Henrik Latter, University of Cambridge
hl278@cam.ac.uk

See images above.

The two images show a snapshot of a numerical simulation of a protoplanetary disc. We are looking at the disc edge-on, and the white dashed lines give the sense of its wedge-like morphology when viewed this way. The background colour on the right panel shows the density of the disc, and from that you also get a sense of the wedge-like disc structure. The central young star (located at the origin) is not in the numerical domain.

The area of interest is in the tangle of magnetic field lines (the black curves on the left panel) and velocity streamlines (the black curves on the right panel). The inner parts of the disc (to the left of the white solid line, which is usually at roughly 1 AU) is undergoing Magnetohydrodynamic (MHD) turbulence because it is hot and sufficiently ionised. The outer part of the disc (to the right) is cooler and not so ionised and thus is relatively laminar. Unexpectedly complex flows and magnetic field structures develop around the transition going from one region to the other. This is probably quite important for planet formation theories, especially those relevant to the Solar System's terrestrial planets (Mercury, Venus, Earth, Mars), which are all located around 1 AU, where all this action is taking place.

Further information about DiRAC and associated RAC calls can be found on the DiRAC website.