Metastable Germanium–tin (GeSn) layers with rather high Sn content between 15% and 18% grown on Si substrates by molecular beam epitaxy were analyzed for the morphological changes on a surface before and after reaching critical layer parameters (thickness, Sn content, and growth temperature) for surface roughening. Atomic-force microscopy investigations were performed as a function of thickness and separately for varying Sn concentrations in the GeSn layer. Epitaxial growth of metastable, uniform GeSn (15% Sn content) layers is obtained up to a critical thickness which increases from about 80 to above 200 nm by reducing the nominal growth temperature from 160 to 140 °C. Phase separation of the complete layer into tin-rich surface protrusions and a Ge-rich matrix takes place beyond the critical thickness. This surface roughening via phase separation was not observed in earlier investigations with lower Sn concentrations (<6%). Tin depletion in the GeSn matrix was confirmed by using energy-dispersive X-ray spectroscopy measurements showing residual Sn concentration below 5%. Additionally, creation of droplets with high concentration of tin on the surfaces was confirmed by energy-dispersive X-ray spectroscopy. Copyright © 2016 John Wiley & Sons, Ltd.