In recent decades, microstructure and texture engineering has become an indispensable factor in meeting the rising demands in mechanical properties and forming behavior of aluminum alloys. Alloying elements, such as Fe and Mn in AlMg(Mn) alloys, affect the number density, size and morphology of both the primary and secondary phases, thus altering the grain size and orientation of the final annealed sheet by Zener pinning and particle stimulated nucleation (PSN). The present study investigates the grain size and texture of four laboratory processed AlMg(Mn) alloys with various Fe and Mn levels (see Part I). Common models for deriving the Zener-limit grain size are discussed in the light of the experimental data. The results underline the significant grain re-finement by dispersoids in high Mn alloys and show a good correlation with the Smith–Zener equation, when weighting the volume fraction of the dispersoids with an exponent of 0.33. Moreover, for high Fe alloys a certain reduction in the average grain size is obtained due to pinning effects and PSN of coarse primary phases. The texture analysis focuses on characteristic texture transformations occurring with pinning effects and PSN. However, the discussion of the texture and typical PSN components is only possible in terms of trends, as all alloys exhibit an almost random distribution of orientations.