Sensory systems display a dizzying complexity of tuning properties and circuitry. Why do neurons in sensory pathways show the properties we observe, instead of the many other possibilities? We investigate whether a simple principle can explain these properties – temporal prediction. This proposes that sensory systems are optimised to efficiently predict immediate future sensory input given recent past input for natural stimuli. This may be useful for guiding future action, uncovering underlying variables, and discarding irrelevant information. We found that simple feedforward networks optimised for temporal prediction on spectrograms of natural sounds produced units with tuning properties resembling those of neurons in primary auditory cortex. When instead optimised for retinal-filtered movies of natural scenes the tuning properties resembled those of simple cells in primary visual cortex. When applied to natural sound waveforms or unfiltered movies, these networks produced units with tuning properties resembling those of neurons in the auditory nerve or retina respectively. Application of temporal prediction hierarchically to the activity of these retina-like units produced units resembling visual cortical simple cells, complex cells and pattern-motion selective cells as one moves up the hierarchy. Furthermore, training a recurrent network for temporal prediction produced functional connectivity motifs between units that resemble those of primary visual cortex, and training a spiking recurrent network for temporal prediction reproduced characteristics of spiking responses in the retina and cortex. Finally, combining recurrency with hierarchical temporal prediction produces properties consistent with those of inter-regional feedback connections in visual cortex. In summary, this simple principle appears to capture many diverse tuning and connectivity properties of sensory neural pathways.