Advanced Computing Facility: EPCC’s secure, resilient data centre

30 years of excellence 

Over the past 30 years EPCC has operated all but one of the UK’s National HPC Services. At a national level, it has also operated HPC systems for the DiRAC consortium and the EPSRC Tier-2 programme. Since 2003, all its services have been operated from its Advanced Computing Facility (ACF) data centre on the outskirts of Edinburgh. 

Additionally, since 2018 EPCC has designed and operated the Edinburgh International Data Facility (EIDF) – a facility focussed on Data Science and AI - the key IT infrastructure for the Edinburgh and SE Scotland City Region Deal. EIDF includes a growing Regional AI GPU Service, which is currently composed of 650 GPU accelerators as a mix of NVIDIA A100, H100 and H200 Tensor Core GPUs. The AI services include a cluster of four Cerebras CS-3 wafer-scale AI accelerators and a Graphcore Pod64 Bow IPU system. 

Investing in future skills 

We are committed to promoting the skills required to develop and operate complex supercomputing and data science systems, and to help young people find their way into careers in our field.

We offer system administration internships and Graduate Apprenticeships for IT undergraduate students via our established partnership with Edinburgh Napier University. Students from the University of Edinburgh’s School of Engineering can join us via industry placements or the University of Edinburgh Employ.ed internship scheme. We are also trialling a Modern Apprenticeship as an alternative route into a career in research infrastructure engineering, and EPCC’s online Masters programme now includes a course in HPC System Administration.

Minimising energy use 

Improving environmental sustainability is a continuous activity. We use a variety of methods to cool our systems, including direct air cooling, semi-direct water cooling, and direct water cooling. Free-cooling, where water is pumped to fans on the roof to be cooled solely by outside air, is used automatically when the outside temperature is lower than the temperature of the chilled water circuits. Variable inlet and outlet water temperatures also maximise efficiency.

A trusted partner 

All our services involve handling user data. We are responsible for ensuring the appropriate level of technical controls are applied to the data to ensure that only the appropriate people can access the data, the data is undamaged, and is available to data owners.

Net Zero and innovation

All our electricity is certified by the Renewable Energy Guarantees of Origin (REGO) scheme, which demonstrates that electricity has been generated from renewable sources only. As a result, the ACF operations are formally categorised as net zero by the Scottish Government.

We are actively researching and developing a digital twin of our data centre to enable modelling and optimisation of our energy and resource usage. The digital twin will help us unlock more efficient system operations, model future systems, and design our machine rooms to maximise efficiency.

The digital twin combines modelling and simulation of power and cooling hardware, along with knowledge of the applications that run on our HPC systems, to predict and explore the interaction between applications running on our systems and the hardware that supports them.

Other innovative projects relating to better use of our excess heat include reducing emissions by sending heat to a Combined Heat and Power plant for use by nearby University of Edinburgh buildings.

We are also part of a £2.6m collaboration exploring the use of flooded mines around the ACF to store and distribute waste heat, with the ultimate aim of making this heat available to homes and businesses.

Technology

The ACF operates a 200 Gbit/s Data Centre Network to support systems hosted on-site and connects to JANET (the UK’s Joint Academic NETwork) at 100 Gbit/s for user access.

As part of the longer-term strategy for the site, capacity has been upgraded from 8MW to 38MW.

Security

We have a commercial-grade data centre firewall from Checkpoint, which operates at 100Gbps.

EPCC holds the following certificates in the internationally recognised ISO standards:

ISO 27001: Information Security
ISO 9001: Quality Management
ISO 22301: Business Continuity and Disaster Recovery.

As part of the management of systems under these ISO processes, we run an active business continuity and disaster recovery (BCDR) process. We continuously test and evaluate our risk management and contingency plans, as well as our standard operating procedures, to ensure service delivery is undertaken in a secure and robust manner

We are accredited for provision of data by the UK Statistics Authority.

Further information

To find out more about the Advanced Computing Facility, please contact us at: info@epcc.ed.ac.uk

Read about the Advanced Computing Facility on the EPCC website.