The larval zebrafish presents an exciting opportunity to investigate the neural basis of vertebrate behavior at the brain scale. However, it has been particularly difficult to distill neural circuits from whole-brain measurements of neural activity. By combining detailed psychophysics, anatomy, cellular resolution whole-brain imaging, and circuit perturbations, we establish critical links between brain- and circuit-level descriptions of the zebrafish optomotor response. Specifically, we find diverse neural response types distributed across multiple brain regions and show that to transform visual motion into action, these regions sequentially integrate eye- and direction-specific sensory streams, refine representations via interhemispheric inhibition, and demix locomotor instructions into distinct motor modules. Ultimately, we develop a quantitative whole-brain model that explains the behavior and reduces the space of possible synaptic connections into a few critical dimensions of functional connectivity among identified neural response types. More generally, our methodology illustrates a flexible paradigm for studying diverse brain-scale computations related to individuality and motivational states.