Although animals can estimate numerical quantities, true counting and arithmetic abilities are unique to humans and are inextricably linked to symbolic competence. However, our unprecedented numerical skills are deeply rooted in our neuronal heritage as primates and vertebrates. In this article, I argue that numerical competence is a cognitive capacity that also determines an individual’s survival and reproduction success. Animals benefit from numerical competence during foraging, navigating, hunting, predation avoidance, social interactions, and reproductive activities. The internal number representations determine how animals perceive stimulus magnitude which, in turn, constrains animals’ spontaneous decisions in the wild. Moving on to the brain mechanisms of numerical competence, I propose that the physiological mechanisms of quantity estimation are part of our evolutionary heritage and can be witnessed across primate and vertebrate phylogeny. Moreover, I suggest that a basic understanding of number, what numerical quantity means, is innately wired into the brain and gives rise to an intuitive number sense, or number instinct. Finally, I argue that symbolic counting and arithmetic in humans is rooted in an evolutionarily and ontogenetically primeval neural system for non-symbolic number representations. These neural constraints jointly determine the basic processing of number concepts in the animal and human mind.