A fundamental question in the literature about value-based decision making is whether values are represented on an absolute, rather than on a relative scale (i.e. context-dependent). Such context-dependency of option values has been extensively investigated in economic decision-making in the form of reference point-dependence and range adaptation. However, context-dependency has been much less investigated in reinforcement learning (RL) situations. Using model-based behavioral analyses we demonstrate that option values are learnt in a context-dependent manner. In RL context-dependence produces several desirable behavioral consequences: i) reference point dependence of option values benefits punishment-avoidance learning and ii) range adaptation allows similar performance for different levels of reinforcer magnitude. Interestingly, these adaptive functions are traded against context-dependent violation of rationality, when options are extrapolated from their original choice contexts.