Recurrent neural networks are an important class of models for explaining neural computations. Recently, there has been progress both in training these networks to perform various tasks, and in relating their activity to that recorded in the brain. Specifically, these models seem to capture the complexity of realistic neural responses. Despite this progress, there are many fundamental gaps towards a theory of these networks. What does it mean to understand a trained network? What types of regularities should we search for? How does the network reflect the task and its environment? I will present several examples of such regularities, in both the structure and the dynamics that arise through training.