Wee Archie production

Author: Guest blogger
Posted: 27 Aug 2019 | 16:32

Caelen Feller, a PRACE Summer of HPC(link is external) (SoHPC) student working with Wee Archie, gives us the final status report of his project. You can also read his original post.

After prototyping my animation server on the Wee Archlet, and creating animations for many simple operations, it was time to get it working on Wee Archie

Transferring my animations to render on Wee Archie was easy in theory… It took less time than I expected, but there were still many complications that need to be ironed out! Below you can see one of my first demonstrations on Wee Archie.

How would you broadcast a message from one computer to every other one on a network? Would you get it to send it to them all, or one by one, or try to send them all at once? No option is perfect – what if the message is large, or the network is slow to start a connection on? A different approach to sending these messages is shown below, which is the best way to make full use of a network where all the computers are connected to each other, and the actual processing of the message will take a while compared to the sending.

Above you can see the broadcast as shown earlier, but broken down to show how it happens inside MPI (a simplified version, anyway). One of the central Pis will send a message to every other one. There are 16 of them so, with the implementation I show, you can do it in four steps! Assuming all your Pis are connected to each other, and all connections are as fast as each other, this is a very fast way of getting a message across.

The central Pi will initially send the message to a neighbour. This means the neighbour also has the message, so they can both pass it on! When they do this to their neighbours, now four have the message. Then they all pass the message on to another neighbour, and then we have eight. If there were more, this would continue. But we’re done after the next step, when the eight each send on their message and all sixteen Pis have it!

This demonstration will be the final one in a series of tutorials building up to it, illustrating the basics of parallel computing on Wee Archie. They include all the previous demos I’ve shown in some form, with explanations and motivation as you go. They’re mostly written and, when the first version is complete, they’ll all be posted on the Wee Archie Github repository.

The main issue with the current system is lack of synchronisation. Because I use HTTP requests to queue all my animations, they often end up out of sync because starting a connection takes time. A potential fix for this is setting up a network of pipes using WebSockets, which I’ll investigate soon. The other main issue is playback speed – I need a global control for frame rate on the various animations, as well as the ability to change this as the circumstance needs it. Hopefully making this part of the server doesn’t prove too difficult, as it would improve the feel of the animations a lot!

Wave simulator overview

The main published goals of my project all discuss the wave simulator, or coastal defence demonstration, but what is it? 

Initially developed by my mentor Dr Gordon Gibb, it’s the most recent demonstration for Wee Archie. It asks the user to place a series of coastal defence barriers – represented here by grey blocks. You have a budget, and each block has a fixed cost. The coastline has several different locations pictured. They are more or less susceptible to being damaged, and damage costs various amounts.

For example, the beachfront uses cheap defences and is easy to damage. The cliff-top housing is hard to damage and expensive to repair. The library, expensive to repair and easy to damage. The shopping market is somewhere in between them all.

Wave simulator (current version)

When you place the barriers and set it running, it will run a fluid simulation and show the waves hitting the shore, calculating the damage they do and how effective your barriers are! At the end of it, the amount of money saved/spent is your score. It’s quite simple to use, and very popular at outreach venues such as Edinburgh Science Festival.

All the current demonstrations use a framework where Wee Archie runs a server on the access Pi (the left of the two on top in the video). This server will run the simulations when called and give the data back to the client, a graphical Python application running on the connected laptop.

Haloswapping & how to show it

Haloswapping in the coastal defence simulation

The main operation used in the coastal defence demonstration is what is known as a “haloswap”. This is an operation at a slightly higher level of abstraction to typical MPI communications, but is very simple. To start you need a grid, or matrix, or array – whatever terminology you prefer.

In the wave simulation, you need to split this grid across many different Pis to run a simple differential equation solver on it (which I won’t go into here). So far, so good. In the video above, we split this grid into 16 equal horizonal strips and give one strip to each Pi (the output from the simulator shows slightly different vertical lines). But how does a wave move from the top of the simulation to the bottom? There is some kind of local relationship in the grid that means changes in one area affect those nearby. How do we pass this local information to other Pis?

The size of this local area of influence is known as a halo, and as I’m sure you’ve guessed from the video, we simply swap the halos between the Pis, known as a “haloswap”. Under the hood, this is a simple send and receive both ways, of an array of numbers (a buffer of data). Once the local information is properly exchanged, the simulation can transfer waves between strips. Wonderful!


Diagram showing a haloswap.

This does have to happen after every iteration (tick of time), so many thousands of haloswaps will occur during the simulation. As this would be absolutely impractical to visualise, there needs to be some kind of limitation on what is shown. For the ease of getting a nice visual output in this project, I’ve just shown every 100th haloswap, a number which would vary depending on the speed of the simulation. Better solutions exist – you could calculate the speed of showing the visualisation vs a sample speed of the visualisation, and show them at the correct rate to keep them in sync, for example, but this is a complex operation given how little it offers in return.

In order to show the haloswap, I added a custom function to my Wee MPI framework (now available in Fortran, as that’s what this and many other HPC simulations are written in), which allows a user to play any named animation on the server, allowing lots of easy flexibility for custom animations such as that shown above.

A web interface?

Something I wanted to try is creating a web framework for displaying simulations because having everything tied to a Python GUI felt a bit restrictive! To recap on the current situation (the framework used for the current version of the wave simulator), all computations are managed by a web server (using Flask) on the top left Raspberry Pi. This Pi dispatches simulations and collects results using MPI, and then continuously returns these results over the network. Currently the client is also a Python application with an interface made using WxWidgets, but this doesn’t have to be the case!

Developed as a NodeJS package, I’ve ported the network code from the client to Python to JavaScript – meaning I can now start to download simulations in the browser. I used React to build the user interface around this and the website it’s embedded in (optimised and built by Gatsby). With this, I’m able to start all of my MPI tutorial demonstrations from the webpage directly and, in theory, also the wave demonstration.

Coastal defence demonstration as a webpage!

As the focus of my project is the visualisation of the parallel communications, I didn’t want to spend too long on web development! I developed a simple version of the whole site, with placeholders in place of some things. For example, while network communications are fully implemented for the MPI demonstrations, handling data in the browser is not the same as in Python!

As a result, I wouldn’t be able to fully re-implement the wave simulation without doing more work than I had time for, though you can see an initial version that allows you to place blocks and shows the benefit of using a webpage as the user interface. Many further features could be added from here, such as a detailed control panel for Wee Archie operable by a student or non-HPC expert, further tutorials and demonstrations, even a virtual version of Wee Archie – WebGL is getting pretty fast nowadays.

Alternatives include a more modern UI framework than WxWidgets, but this requires a lot more knowledge than web development of the average Wee Archie developer. I’ll leave the final decision on the path to take up to the maintainers of Wee Archie here at EPCC!


I’d like to thank everybody here at EPCC and PRACE for such a wonderful experience, in particular my mentor, Dr Gordon Gibb. I’ve learned a lot, from MPI to video editing, in these two months and I’m sure I’ll never program in serial again if I can help it.

All my code will soon be available on the Wee Archie Github repository. I’ve embedded my final video presentation below. My final report will also be available on the Summer of HPC site. If you’d like to follow my work in the future, check out my website!

I hope you’ve enjoyed my blog posts. Goodbye for now!


Blog Archive