To uncover the effects of segregation and precipitation on grain refinement of metals under severe plastic deformation, high-pressure-torsion (HPT) processing is performed on three binary Al-(Si, Mg, and Cu) alloys with different stability and segregation characteristics. Atom probe tomography analysis reveals that HPT processing induces significant decomposition of Al-1 at% Si and Al-1 at% Cu alloys with coarse Si particles and fine Al ₂Cu (θ) precipitates formed, but no decomposition of Al-1 at% Mg alloy, with dislocations segregated with Si, Cu, and Mg, and grain boundaries (GBs) only segregated with Cu and Mg. The GB segregation of Cu is stronger than that of Mg and Si, with Cu excess in the range of 1.5–7.0 atoms/nm ², Mg excess in the range of 0–4.0 atoms/nm ², but no Si excess. Interestingly, some GBs without Mg segregation develop a Mg-depletion zone along a single side. All evidences demonstrate that GB segregation and precipitation are responsible for HPT-induced grain refinement of Al-1Mg and Al-1Cu alloys but coarsening of the Al-1Si alloy. Engineering solute distribution is of significance in controlling the ultrafine grain of the Al alloys.