This document is intended for those applying to the HPC-Europa Transnational Access programme to visit EPCC in Edinburgh, UK. Please read this document carefully before closing your application form and contact us (europa@epcc.ed.ac.uk) with any questions you may have. It is also hoped that this document is useful to visitors who have already been accepted to visit EPCC and are waiting for their visit to start.
Hardware
At EPCC there are principally three parallel computers available for HPC-Europa visitors:
- A 32-processor AMD-based, Opteron system (Ness).
- A 12,288-processor XT4 system (HECToR Phase 2a).
- A 44,544-processor XT6 system (HECToR Phase 2b).
Quantifying resources
Please note there are two units used when quantifying resources on our parallel computers and those of the other HPC-Europa centres. The first is simply the processor hour (or CPU-hour) which is literally the number of processors used multiplied by the number of hours for which an application runs. The second is the Allocation Unit (AU) which is calculated by taking the number of processor hours and multiplying this value by a factor (the AU rate) which is dependent on the speed of the machine when tested running a parallel code. The AU rate is taken from the performance of the Linpack code and it is the rate used when establishing the position on the Top 500 fastest machines in the world. HECToR Phase 2b (XT6) is presently 16th in this list. You should note that this rate is based on just one code and your code may not achieve the same performance. However, all time used on our machines is reported to the EU in AU using the rates indicated below1. HECToR has work disks which can be used as storage or scratch space, in addition to smaller, backed-up home directories.
Ness
By default all visitors get access to Ness and this can be used for testing and debugging code as well as for small to moderately large production runs. This machine is suitable for jobs of up 12 hours on 2-16 processors. The processors in this machine are each 2.6 GHz AMD Opteron (AMD64e) processors with 2 Gbyte of RAM. This is a “shared memory” machine and is suitable for running OpenMP codes as well as MPI and multi-threaded codes.
The AU rate on Ness is 1 processor hour = 4.4 AU.
HECToR
By default all visitors get access to HECToR. Visitors should note that they may be required to show evidence of the efficiency of their codes when running on large numbers of processors, before they can gain access to significant resources on HECToR. The reason for this is because our resources are limited (see “Amount of resources available to visitors” section below).
The AU rate on HECToR Phase 2a (XT4) is 1 core processor hour = 7.5 AU.
The AU rate on HECToR Phase 2b (XT6) is 1 core processor hour = 3.77 AU2.
Amount of resources available to visitors
Our resources on these machines are limited in different ways: At EPCC we expect to accommodate just over 50 visitors per year and we have a total of approximately 15,000,000 AU available for 2011 (approx 2,000,000 CPU-hours). This implies around 40,000 CPU hours per visitor. However, the amount of resources available depends on the kind of jobs you will be running and the overall usage of the machines. Of course many of our visitors will consume only a small amount of resources if their visit involves the early, development stage of their application. For this reason we can often give out much more to some visitors than simply giving the same “average” value to each.
You should bear in mind the above figures when applying to the programme. As well as considering the amount of resources available in total you should also consider the throughput of your jobs. For example, if you are running 12 hour, 16-processor jobs on Ness you would not expect more than 1 of these jobs to run per day. Consequently burning 20,000 CPU-hours would take over 100 days! We allocate resources on a case-by-case basis and if in doubt you should contact us (europa@epcc.ed.ac.uk) before completing your application form. If your request requires more than 100,000 CPU-hours then it is very important that you contact us to discuss these requirements before the visit starts and you must realise that we may not be able to fully satisfy these requirements.
Filling in the “Resources” section of your HPC-Europa application form
We appreciate that it can be difficult to make an estimate of how much time your project will require, especially if your code is not yet parallelised. For this reason it is useful if you state the assumptions you have made when estimating the resources. For example, are your estimates made on the basis of how long the serial code takes to run? On what architecture (make, processor speed) does your code usually run? Are you assuming perfect speed-up of your code (i.e. that it will run p times faster on p processors)? If your estimates are based on code which is already parallelised and has been tested then please give details of these tests on the form. The more information you give us and the more accurate the information is, the better we will be able to assess your application.
Acceptance of your application
Please note that acceptance of your application does not automatically mean that you will have access to exactly the resources you have specified in the application form, particularly if you have specified a large amount of resources. The reason for this is that we cannot reserve very large allocations based just on your application form until we are sure that you will use it. Doing so would prevent others from performing their work. For this reason it is very important that for large requests for time, you contact us before your visit (preferably before you submit your application). This way we can get an agreement of the amount of resources you will use in advance of your visit.
Length of access to resources
Visitors are usually given access to Ness, and possibly HECToR, before they arrive in readiness for the visit. However, you will not be given significant resources until your visit commences. Remote access is usually granted for 4 months after a visit has ended in order to finish work which may not have been completed during a visit. Please note that we are unable to provide large amounts of resources to visitors applying for a very short visit (e.g. ~2 weeks). In general we have found that longer visits (e.g. 6 weeks or more) have usually been more productive.
Access to other machines…
We do not presently have direct access to any machines other than those stated above. You should build time into your workplan (e.g. ~1 week or more for a complicated code or one with lots of dependencies of libraries or particular compilers) for porting your code. If your codes are written using standard languages/libraries this should be possible and we can help with this. As a general rule you should try to avoid writing codes in such a way as they rely on the features on one particular compiler/builder.
Software
Serial/Parallel codes
It is essential that your application to HPC-Europa is based on HPC. This basically means that you need to have a requirement to run your codes on parallel architectures or be doing research in the area of HPC itself. This does not mean that you need to have a parallel code before your visit starts, indeed very many visitors use their visit to parallelise their codes, but you must at least be in a position where your code can be parallelised during your visit and you should investigate the possibility of success in this task before applying to HPC-Europa. We cannot accommodate visits where the intention is simply to run serial codes and our machines are not equipped for this purpose.
Packages
If you are using a third-party package or you are required to link your code to third-party libraries you must check in advance that our machines either have these packages/libraries installed or that you are sure that they can be installed. We can usually help out with the installation of such software but you must check that the software is compatible with the machine you wish to run on. It is also important that you check the licensing arrangements for third-party software. Just because we have software installed does not mean you are licensed to use it. Information of third-party packages, libraries and associated licences is available on the websites of the various machines.
Matlab
Please note that we do not have a Matlab licence and Matlab is not installed on any of our machines. If you require access to Matlab then you will need to contact your host to check that he/she can give you access to it in the host department and the use of Matlab cannot be the main purpose of your visit as it will not run on any of our HPC machines. If Matlab is used to perform your computational work then you will need to consider whether or not your application is really an HPC application (i.e. an application with can be run on parallel computers). We cannot accommodate applications consisting simply of serial runs of Matlab and in order for your application to be accepted you will need to research the best method to produce a parallel code from your Matlab code. Our investigations show that this is not usually straightforward.
Footnotes
- 1. Note that these rates will change as machines are upgraded.
- 2. Note this figure is presently a discounted rate. After the interconnect upgrade around April 2011 this is likely to increase.