The ability to understand other people’s actions is a fundamental basis for social interactions. According to a dominant view in the literature, this ability critically relies on the recruitment of parietal and frontal regions that are also involved when we plan and perform actions ourselves. Here I will present a number of recent studies using multivariate pattern analysis (MVPA) and representational similarity analysis (RSA) of functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data to identify action representations that generalize across action properties that are irrelevant for action understanding. I will discuss the results, which highlight the importance of the lateral occipitotemporal cortex for such abstract representations, in light of the ongoing debate on the neural basis of action understanding, and point out future research directions.