Climate change is expected to reduce agricultural productivity in developing countries. Classic international trade and geography models predict that the optimal adaptation response is a reallocation of capital and labor from agriculture towards sectors and regions gaining comparative advantage. In this paper, we provide evidence on the effects of recent changes in climate in Brazil to understand to what extent factor market frictions constrain this reallocation process. We document that persistent increases in dryness do not generate capital reallocation but a sharp reduction in credit to all sectors in both drying areas and financially integrated regions. In addition, dryness generates a large reduction in agricultural employment. Workers staying in drying regions reallocate towards manufacturing but climate migrants are allocated to small firms outside of manufacturing in destination regions. The evidence suggests that frictions in the interbank market and spatial labor market frictions constrain the reallocation process from agriculture to manufacturing.