Supercomputer driving tests
Posted: 16 Dec 2014 | 11:27
In 2013, the DiRAC consortium rolled out the DiRAC driving licence, a software skills aptitude test for researchers wanting to use DiRAC's high-performance computing resources. Now ARCHER, the UK National Supercomputing Service, is to roll out an ARCHER driving test.
Despite their similar names, these tests differ in nature, intent, scale and reward. In this post, EPCC's Mike Jackson, Andrew Turner and Clair Barrass compare and contrast these two supercomputer tests.
DiRAC Driving Licence
DiRAC is the UK's integrated supercomputing facility for theoretical modeling and HPC-based research in particle physics, astronomy and cosmology.
The DiRAC driving licence was originally developed in 2012 by Mike Jackson as part of The Software Sustainability Institute; Greg Wilson of Software Carpentry; and, from DiRAC, Andy Turner and James Hetherington of the Research Software Group at University College London; DiRAC project director Jeremy Yates, and project manager Harpreet Dhanoa.
The driving licence covers software development skills including the shell and automation, version control, testing, code review, the use of public/private keypairs and secure shell. DiRAC users sit an open book test (a sample test is also available for them to look at in advance), which takes 1-1.5 hours, under the supervision of a trainer. The trainer marks each user's answers, providing one-to-one feedback via email or Skype.
The licence is mandatory for all UK-based DiRAC users who are studying for advanced degrees, such as MSc and PhD, and for those employed as post-doctoral research assistants/associates with less than 3 years’ experience. Other users are also encouraged to take the test.
Upon successful completion, users are awarded a certificate of competency. Those who fail are directed to courses to fill in the gaps in their knowledge, before sitting the test again at a later date.
The driving licence is intended to encourage users to undertake training which will help to improve their scientific computing skills and so to deliver research that makes a greater impact. It also helps the DiRAC consortium to ensure its resources are used as efficiently as possible.
The driving licence was rolled out across the DiRAC consortium in 2013, and has been run numerous times at seven sites across the UK, including here in Edinburgh where it was run by EPCC's Nick Brown. The driving licence will continue to run throughout 2015.
ARCHER Driving Test
ARCHER is the UK National Supercomputing Service. ARCHER provides a capability resource to allow researchers to run simulations and calculations that require large numbers of processing cores working in a tightly-coupled, parallel fashion.
The ARCHER driving test has been developed by the ARCHER Computational Science and Engineering (CSE) team here at EPCC. The test consists of questions that cover both ARCHER's resources (eg “ARCHER uses which type of processor?”) and how day-to-day development is done within ARCHER's infrastructure (eg “What are the default names of the files into which the job submission system places the output from a job?”). The test consists of multiple-choice questions which either require users to choose a single answer or a set of answers. The test is taken online and feedback is provided from pre-prepared responses that provide more information about both correct and incorrect answers, with links to further information. The questions, and supporting commentary, were crowd-sourced from within the CSE team.
Users who pass the test are given time on ARCHER as a reward. The test is optional and is intended to provide another, lightweight, avenue by which users can get access to ARCHER's resources (e.g. users with experience of other HPC systems who can demonstrate they can make effective use of ARCHER). The test can also be taken by attendees at ARCHER’s Hands-on Introduction to High Performance Computing courses to help them to identify what they have learned and what they might need to revise. It is hoped that rewarding success with time on ARCHER will encourage attendees to apply some of their new learning immediately, while still fresh in their minds.
The ARCHER driving test will be rolled out in 2015.
Scalability versus personalisation
The ARCHER driving test is less time-intensive and more scalable than the DiRAC driving licence, but at the expense of personalised feedback on each user's skills and training requirements. However, such comparisons need to be viewed in light of each consortium’s user communities and training services.
DiRAC has ~500 registered users. Most are permanent academics or PDRAs with more than 3 years’ experience so are exempt from taking the driving licence. DiRAC provides no training courses of its own, rather it recommends a portfolio of online material that DiRAC users are encouraged to study. The driving licence serves as a motivator for users to undertake this study. Personalised face-to-face feedback from trainers provides users with the opportunity to ask questions and resolve problems that would, in other contexts, be resolved on training courses.
In contrast, ARCHER has ~3000 registered users. To deploy a DiRAC-style driving licence across such a large community, with personalised marking and feedback would be un-scalable. However, ARCHER runs a large number (32 in 2014) of training courses (at introductory, intermediate and advanced levels) at sites across the UK, complemented by virtual tutorials, which offers scope for users to ask questions and resolve problems specific to them.
Though the DiRAC driving licence and ARCHER driving test differ in both nature and delivery they share one important quality: they both encourage the users of these HPC resources to improve their understanding of the resources and how best to use them. As a consequence, they both help to contribute to the development of users' skills in software development, and also in the more effective and efficient usage of HPC resources which are in heavy demand.
Image: David, Flickr