Recovery of cold worked metals is associated with a loss of strength due to the reduction of the defect density. However, already in the 1960s analytical models and few experiments suggested that this may not generally be the case and even a hardening rather than a softening might occur. With the availability of severe plastic deformation and deposition techniques this anneal hardening phenomenon has been observed frequently. In this overview we summarize early findings on this topic before focusing on general observations and potential origins, with a special focus to show the similarities for nanostructures across different grain size scales (i.e., structures prepared by deposition and severe plastic deformation techniques, respectively). Comparison of different results indicate a grain size dependent hardening increment that could be additionally affected by segregation or the processing variables. Considering the agreement of the peak hardening temperatures with that for dislocation annihilation at grain boundaries, anneal hardening can be rationalized by the loss of intragranular defects and grain boundary relaxation. Grain boundary diffusivity hence plays a crucial role and particular solutes could amplify the hardening process even further. As anneal hardening already occurs for grain sizes at the micron scale, its effect on properties is even of technological relevance. Beneficial effects on the fatigue strength are evident, but the strain softening of anneal hardened specimens drastically shorten ductility. Nevertheless, some strategies to overcome this adverse effect seem promising to create metallic structures with exceptional combinations of strength and ductility.