The recrystallization behavior of five low alloyed steels is investigated using double hit deformation tests. It is shown, that Niobium has the biggest influence in retarding the recrystallization kinetics. Further, the microstructural evolution dependent on strain and temperature during deformation is studied with a picric acid etchant and light-optical analysis. It is shown how the microstructure of two differently alloyed ultra-high strength steels changes along with the peculiarities of the corresponding stress–strain curves including the evolution of grain size and aspect ratio of the prior austenite grain. The findings on the different recrystallization kinetics with the role of recrystallization retarding elements are further reinforced by investigations on the Zener-Hollomon parameter and the activation energy needed for dynamic recrystallization. A rolling scenario on a deformation dilatometer is simulated on a hardenable and a micro-alloyed steel to illustrate the microstructural evolution between the rolling steps. It is shown, how the two ultra-high strength steels perform different in their microstructural evolution, as the waiving of micro-alloying elements (MAE) provides finer austenite grains.