The application of covert visual attention has clear behavioral effects: it can increase accuracy and reduce reaction time on challenging visual tasks. Neural recordings performed while animals execute these tasks have captured several neural correlates of attention. Questions remain, however, regarding how the observed neural correlates relate to the observed behavioral changes. Utilizing the ability of deep convolutional neural networks to perform visual tasks, we explore the impact that biologically-inspired attentional modulations have on performance in challenging visual tasks that require feature and spatial attention. Interestingly, the modulations that most closely match the biology perform best in this artificial network. We also build a more biologically-realistic version of a CNN that incorporates recurrently-connected E and I cells at each convolutional layer. This model also replicates the behavioral effects of applying attention, and allows for a more direct comparison between neural data and the model.