It is often said that primates are “visual animals”, and the primate neocortex has dozens of visual areas to confirm this impression. Yet some of the most important contributions that vision makes to the evolutionary success of primates depend on the prefrontal cortex. The ventral visual processing stream has axonal connections with two major parts of the primate prefrontal cortex: the ventral prefrontal cortex (PFv) and the orbitofrontal cortex (PFo). To understand their contributions to visually guided behavior, our laboratory uses either permanent, selective lesions or temporary inactivations of these areas in macaque monkeys. Using the 3-arm bandit task, we found that PFv is necessary for rapidly updating availability valuations of visual stimuli. These valuations change to reflect the current likelihood that the choice of a visual stimulus will lead to a desirable outcome (such as food availability). In contrast, PFo is necessary for updating desirability valuations of the outcomes predicted to follow the choice of either an object or an action. This function depends on tracking the current value of food items (outcomes) as they become devalued during selective satiation, and it requires interactions between PFo and the amygdala. PFo’s various components make different contributions in this regard. A posterior part of PFo, area 13, is necessary for its value updating function, while an anterior part, area 11, is necessary for translating these valuations into actions. We have also found that monkeys with PFo lesions fail to acquire a Pavlovian arousal response to visual stimuli that predict reward, although these monkeys can learn instrumental (saccadic) responses normally for the same kind of stimulus material. These data provide insight into the role of PFv and PFo in reward-based decision making and, perhaps more importantly, they specify the contributions that the two areas make to the adaptive fitness of primates.