Quantifying the climate impacts onto economic outcomes is crucial to inform mitigation and adaptation policy decisions in the context of anthropogenic climate change. Projections of costs and benefits of possible mitigation pathways are frequently created using Integrated Assessment Models (IAMs), which inherently rely on damage functions that relate warming levels to associated economic damage. These damage functions, however, have very little grounding in empirical evidence. In turn, empirical studies of climate impacts frequently use highly aggregated and simplified measures for climatic conditions, such as country-averaged annual average temperatures, possibly masking much of the sub-annual extreme weather events that can greatly affect society. Here we estimate the impact of climate onto economic growth using a large set of candidate climate variables, including measures for extreme events in a global panel dataset. Using econometric model selection methods robust to outlying observations, we identify relevant climate variables without a-priori imposing their inclusion in the model. Results suggest that economic impacts are more accurately described by supplementing commonly used temperature indicators with extreme climate indicators, although their specification is not fully robust to the selection method. We project future climate impacts onto economic growth using SSP scenarios and CMIP5 projections to construct an empirically-derived damage function based on the identified relationship, allowing for adaptation over time. Our estimates of an empirical damage function uncover large discrepancies when compared to damage functions currently being used in IAM projections. The use of an empirically-derived damage function increases DICE maximum estimates of the Social Cost of Carbon by nearly an order of magnitude and increases SCC estimates for 2050 from about US$90 to over US$600.