Presenting My Paper at Euro-Par 2025

Last week I was in Dresden for Euro-Par 2025! I went to present my paper about speeding up neural network training with a new kind of parallel thread synchronisation scheme. It was the first time doing that since starting my PhD back in September 2023, so I wasn’t quite sure what to expect, but it was a really fun event. I even managed to sneakily record my own presentation (see below!)

The Paper

The paper I presented at Euro-Par was titled “Interval-Asynchrony: Delimited Intervals of Localised Asynchrony for Fast Parallel SGD”. It’s the first paper I managed to write and get published during my PhD (so far!)

Watch the 20 minute presentation – or even better, read the paper – for more details, but here’s an overview (simplified, but should still get the point across). Basically, the problem is that we wanted to use the stochastic gradient descent algorithm (SGD) to train a neural network, which in our case is being used for image classification. If you want to speed up this algorithm, you can run it in parallel, which just means that you split up the work between a number of workers, which in our case are CPU threads.

The SGD algorithm consists of a number of iterations, and each iteration depends on the result of the previous one. At least, it does if you want it to be completely accurate. Then, the natural way to parallelise this process is to split the work of each iteration across all your threads, wait for them all to finish their piece, and then combine those results together somehow. Don’t worry about the details, other than what I’ve already explained.

So, this works, but some threads are inevitably at least a tiny bit faster than some others, and so they must spend some time doing nothing while waiting for slower ones to finish. This discrepancy adds up when you use hundreds of threads, and significantly limits the speed at which you can train your neural network.

The other side of the spectrum is asynchronous execution, where you basically just let all the threads run iterations independently, with no synchronisation. They all see the same neural network parameters in memory, and can just update them whenever they want. It’s really fast in terms of iterations per second, but significantly less accurate than synchronous execution, so maybe more iterations are needed to train the model.

We (and by we I mean my PhD supervisor Philippas Tsigas and I) thought that this trade-off was interesting, and designed a new type of execution scheme which has a parameter to determine “how asynchronous” you want it to be. This is really useful, because actually it turns out that as you get closer to convergence, you want more accurate updates, and so you can just change this parameter over time!

In terms of how exactly the amount of asynchrony is enforced: the idea is that we divide the whole execution into a series of “asynchronous intervals”. Threads run normal asynchronous execution during an interval, but any iterations that overlap two intervals are just ignored, i.e. not used to update the model. An interval ends when a certain number – that parameter that I mentioned a minute ago – of iterations have been completed during it. The point is that we can bound the inaccuracy by keeping this interval small, or we can widen it and speed up the throughput by rejecting fewer steps. Again, please look at the paper for proper details.

Anyway, it turns out that this is a pretty nice way of adjusting the amount of asynchrony, which is useful when paired with a clever way of setting that interval size parameter (which we also did in the paper!)

The Conference

As I mentioned, it was my first proper conference, so I didn’t really know what to expect. I was quite looking forward to presenting my paper, and so I spent a couple of the evenings after the conference practicing it a bit. Weirdly – and this seems to happen in general – the more I rehearsed it, the more I forget what to say while rehearsing, but it always seems to be the case that once I actually go to present it for real it goes fine, more or less.

The conference itself was really well organised, and I met some really nice people there. In fact, I didn’t expect I would manage that much networking and socialising – it’s not something I do very much – but even on the first day (which was workshops rather than the main conference) I met a couple of nice people who were also going to be presenting their work. I think the fact that Euro-Par’s scope is quite broad (parallel programming of all kinds, really) made it a lot more interesting: if every presentation was in the same niche as mine, then it might get a little boring? But the theme at least meant I had some kind of understanding of the other works, at some level.

Hopefully I’ll get another paper accepted there in the next few years so I can go back :)

Written 03 Sep 2025 by Jacob Garby