The in vivo responses of dorsal raphe nucleus (DRN) serotonin neurons to emotionally-salient stimuli are a puzzle. Existing theories centred on reward, surprise, salience, and uncertainty individually account for some aspects of serotonergic activity but not others. Building on recent insights into the physiology of the DRN, here we find a unifying perspective in a prospective code for value, a quantity central to reinforcement learning theory. Our model explains why serotonin neurons are activated by both rewards and punishments, and why these neurons are more strongly activated by surprising rewards but have no such surprise-preference for punishments—-observations that previous theories have failed to reconcile. In addition, our model quantitatively predicts in vivo activity fluctuations on the order of a single spike per trial that may have escaped notice in previous work. By unifying previous theories and establishing a deep connection with reinforcement learning, our work represents an important step towards understanding the role of serotonin in learning and behaviour.