File Transfer

As ARCHER and the UK Research Data Facility (RDF) are both coming to the end of their service, many users will have to transfer their data to their own machines. Because of this, the ARCHER Data Transfer Guide has been updated to give more precise and up-to-date information about best practices for transferring data between ARCHER, the RDF, and remote machines.

Introduction

While a surprisingly high proportion of HPC users are happy to keep their data on a single HPC service, or at most to move it within the hosting institution, sometimes is becomes necessary to move large volumes of data between different sites and institutions. As anyone who has ever tried to support users in this endeavour knows, it can be much harder to get good performance than it should be. This post is an attempt to document the available tools and technologies as well as common problems and bottlenecks.