The International Congresses of Mathematicians (ICMs) have taken place at (reasonably) regular intervals since 1897, and although their participants may have wanted to confine these events purely to mathematics, they could not help but be affected by wider world events. This is particularly true of the 1936 ICM, held in Oslo. In this talk, I will give a whistle-stop tour of the early ICMs, before discussing the circumstances of the Oslo meeting, with a particular focus on the activities of the Nazi-led German delegation.