Summer of HPC: Week 3
Posted: 19 Aug 2015 | 14:20
Jana Boltersdorf is visiting EPCC as part of the PRACE Summer of HPC programme. Together with Ondrej Vysocky she has been working on a project that is Developing the user interface for the Fluctuating Finite Element Analysis (FFEA) tool.
Jana has been working on introducing MPI parallelisation into the code which currently only scales to 8 cores using OpenMP, while Ondrej has been developing the user interface for FFEA, in particular the viewing tool, which currently is limited by memory and lack of threading for looking at large datasets.
Here Jana describes her third week of the Summer School.
The first and probably most important task due for our third week in Edinburgh was the Week 3 report. It’s not only about presenting our project after about two weeks of work but also updating the time schedule and preparing the final report. The finished Week 3 reports will be compiled into one document and be available to the public in the near future.
I had already started it during the previous week as it was due on Tuesday morning of the third week but, of course, there was still some work to do. Adding pictures, do some proof-reading, adjust the formatting, re-phrase some sentences, and remove redundant statements. However, just after I handed in the final version on Monday, the submitted time schedule started to fall apart slowly.
In the previous weeks, I had four different test cases but only one of them included some actual interactions between the simulated molecules: Two spheres slowly approaching each other, bumping into each other and then drifting apart slowly. As you can easily imagine, that’s a pretty simplistic test and far away from any realistic application of FFEA – a tool for simulating large numbers of proteins. Also, at this scale, the current parallelisation had hardy any noticeable effect on the runtime no matter whether I used the “per-blob parallelisation” (pb) or the “with-blob parallelisation” (wb).
As a result, I asked Albert and Ben (the people at the University of Leeds who wrote the code that Jana and Ondrej have been working on) for a more realistic test to study the performance of the current parallelisation approaches.
They reacted promptly by providing a larger example: Seven “blobs” (the shapes that represent the proteins) with Van der Waals interactions. In fact, it was so large that the simulation still wasn’t finished after more than three days of calculation. That kind of of runtime is not very useful in the early stages of a project where you have to do lots of runs to get an idea of how the codes work and which approaches you should pursue! So, we tried to simplify the test to reduce the runtime – down to four blobs and less time steps. My goal was a somewhat realistic test that still completed in less than an hour.
As I kept changing the different parameters of the simulation, I witnessed an odd behaviour: For one million time steps, the program finished within just a moment, for ten millions it kept running for several hours until I cancelled the calculation. Calculation time is rarely a linear function but this seemed quite extreme!
I found the reason quite accidentally – I just turned off the Van der Waals interactions to study their impact. Doing so allowed me to disable the restart feature as well. And suddenly, the runtimes behaved exactly as expected. As it turned out, the restart feature looks at the output trajectories and starts at the last completed trajectory frame. The test file I had received contained trajectory output from previous runs for the first million time steps and therefore didn’t do any calculation up to this point. That’s why it was so fast. Without the restart, it took pretty much exactly an hour – exactly what I had been looking for all the time.
Quite an easy explanation, maybe even obvious now, but before I knew about the restart feature and the old trajectories, it puzzled me a lot! Now, I was finally able to run the test for the different parallelisation strategies and get an idea of how they performed.