Simulation code usage on HECToR

Author: Andy Turner
Posted: 24 May 2013 | 16:00

What sort of research is the HECToR supercomputing facility used for and what simulation software does it make use of?

EPCC measures the use of different simulation codes used on the HECToR facility to get an idea of which codes are used most and what size of jobs different code are used for. In this post I will take a look at which codes are used most on the facility and speculate whether we can infer anything from the patterns we see.

The top 5 applications used on the HECToR Phase 3 system (from November 2011 to now) rated by total core hours account for almost 40% of the total time used on HECToR. They are:

Application Core Hours % Total Science Area
VASP 135,684,992 19% Materials Science/Chemistry
CP2K 56,897,504 8% Materials Science/Chemistry
GROMACS 40,123,232 6% Biomolecular Science
DL_POLY 34,498,112 5% Materials Science
CASTEP 30,408,544 4% Materials Science/Chemistry

It is obvious that the current time use on the HECToR facility is dominated by the materials science and chemistry codes. Why is this so? There are a number of contributing factors, possibly including:

  • The use model is more efficient in using time on the system.
  • They are more successful, relatively, in getting HECToR time awarded for their research.
  • The large computational physics communities - astrophysics and particle physics - are not present on HECToR (they have their own national facility: DiRAC).
  • Generally, the codes scale to larger numbers of cores than other areas so they are able to use large amounts of time on large HPC systems.

Use model

In contrast to many other computational communities, the vast majority of scientists in this area are code users but not code developers. The researchers are free to expend their effort using codes provided by others to perform scientific research rather than spending much of their time developing code. This model of separation of researchers and developers (with the developers still being part of the research community) is also used by the climate modelling community on HECToR who also use a large amount of time on the system.

Time awarded on the system

From the HECToR reports (http://www.hector.ac.uk/about-us/reports/) we can extract the following distribution of time awarded on the system:

  • Chemistry - 50%
  • Earth Sciences - 21%
  • Engineering - 18%
  • Materials - 4%
  • Physics - 6%
  • Life Science - 1%

With the pattern of allocation on the system, it should be no surprise that the most used codes are from the chemistry area. The results are also skewed by the absence of particle physics and astrophysics applications from HECToR: the usage pattern is not representative of UK computational science because a large fraction of the community is absent.

Job size distribution

The job size as a function of compute time used for the five top codes is shown in the plot below. There is actually little difference between this plot and the same plot for all usage on HECToR. This suggests that these codes (or, at least, the problems being studied using these codes) do not scale any better than other codes and scientific problems studied on HECToR.

The only real exception to this is the DL_POLY code which has the largest fraction of its usage at the largest standard job size supported on HECToR (65536 cores, 2048 nodes).

What does this mean for the future?

Here are a few questions that the pattern of code usage suggested to me. I am sure that there are more - feel free to share any of yours below.

Apart from GROMACS, none of the top 5 codes can exploit accelerators in any meaningful way. Is this going to be a problem for these researchers in future, given that most commentators think that vector/SIMD architectures are going to be increasingly important in HPC? Are there features of these codes that place limits on the amount of parallelism available at the processor level?

Should we be focussing our software development effort on these types of codes to make sure they are able to exploit upcoming HPC architectures? Or, as these communities seem to be doing well in using large amounts of computer time, should we concentrate on enabling applications from communities that are not yet managing to exploit the resources? This does not necessarily mean that we should exclusively invest in these particular codes - there may be other factors at play in terms of availability of the code to work on, suitability for development etc that provide input into the codes we should be working on. Nevertheless, this information does provide useful input into that discussion.

It is also interesting that codes 2 and 3 on the list (CP2K and GROMACS) are both open source projects. This level of utilisation of open source software would not have been seen even five years ago. Does this indicate a trend for the future?

Of course, as well as total statistics such as these we could also look at time-dependent trends: is the use of a particular application (or type of application) growing? Maybe this will be the focus of a future blog post!

What do you think? Please add your comments below - I would be interested to know your thoughts.

Appendix: technical details

We use an extremely naive method for assessing what applications people use on the system:

  • The currently running applications are polled every hour and the following details of all running applications stored:
    • Job ID
    • Job size (in nodes)
    • Username
    • Executable name

This data can then be analysed:

  • The executable names are compared against a list of known names using regular expressions (regexp, http://www.regular-expressions.info/).
  • Any unknown executable name is captured and used in future regexp matching to capture usage of codes that are not known a priori.

The results of the analysis give us an overview of what applications are using significant time on the facility and the distribution of job sizes as a function of time (and jobs submitted).

Further Information

The code usage statistics for HECToR are published publicly as part of the annual reports on the HECToR website. The reports also contain a large amount of other information relating to the use of the facility. For more details, please see:

Contact

Andrew Turner, EPCC