Deep convolutional neural networks (CNNs) provide a potentially rich source of insight for understanding mid-level visual processing in the primate cerebral cortex. Taking the approach of an electrophysiologist to characterizing single CNN units, we found that many units exhibit translation-invariant boundary curvature selectivity approaching that of the best neurons in the mid-level visual area V4. For some of these V4-like units, particularly in the middle layers, the natural images that drove them best were also qualitatively consistent with selectivity for object boundaries. Our results identify a novel image-computable model for V4 boundary curvature selectivity and suggest that such a representation may begin to emerge within the middle layers of an artificial network trained for image categorization, even though boundary information was not provided during training. This raises the general possibility that single-unit feature selectivity learned in CNNs may become a valuable guide for understanding sensory cortex.