Understanding composition effects is crucial for alloy design and development. To date, there is a lack of research comprehensively addressing the effect of alloy composition on dynamic precipitation, segregation and grain refinement under severe-plastic-deformation processing. This research investigates Al-xSi alloys with x = 0.1, 0.5 and 1.0 at.% Si processed by high pressure torsion (HPT) at room temperature by using transmission electron microscopy, transmission Kikuchi diffraction and atom probe tomography. The alloys exhibit interesting composition-dependent grain refinement and fast dynamic decomposition under HPT processing. Si atoms segregate at dislocations and Si precipitates form at grain boundaries (GBs) depending on the Si content of the alloys. The growth of Si precipitates consumes most Si atoms segregating at GBs, hence the size and distribution of the Si precipitates become predominant factors in controlling the grain size of the decomposed Al-Si alloys after HPT processing. The hardness of the Al-Si alloys is well correlated with a combination of grain-refinement strengthening and the decomposition-induced softening.