Humans and other animals are highly adept at adapting their behavior to changes in the rewards in their environments. Trial-and-error learning is one of the simplest mechanisms for such improving from experience and forms the backbone of many of the key computational models of reward learning. These models, however, are unable to explain even basic learning phenomena, such as spontaneous recovery, where the degree of responding in a learning task can change with the passage of time. In this talk, I present a computational model that extends trial-and-error to learning from imagined or replayed experiences. This extension provides a novel explanation for many puzzling aspects of conditioning as well as for memory biases in risky choice and even potentially for curiosity and information-seeking. The breadth of empirical phenomena addressed by the model illustrates the power of a trial-and-error learning rule applied to both real and remembered experiences.