The effects of carbon and residual sulfur on the grain boundary (GB) mobility in austenite were studied by in situ experiments, density functional theory (DFT) and mean field simulations. Fe-C samples with carbon contents ranging from 0.15 wt.-% to 0.46 wt.-%, containing impurities S (<25 wt. ppm) and P (<25 wt. ppm), were investigated under isothermal annealing conditions at 1050–1350 °C. High-temperature laser scanning confocal microscopy was used to observe and quantify in situ isothermal grain growth. The results demonstrated significant variations in grain growth kinetics and final grain size depending on C content and temperature. Above 0.25 wt.-% C, grain growth increased markedly, potentially due to increased GB segregation of C. To rationalize the experimental observations, a multiscale modeling workflow combining atomistic DFT calculations with mean field simulations of grain growth was used. The energy profiles of solute C and impurities S and P were determined for two different GB types (sigma5 and sigma13). The segregation analysis revealed that C competes with P and S for grain boundary sites. Mean field simulations of GB enrichment and GB migration using DFT data provide an explanation for the increase in grain boundary mobility in alloys with sufficiently high C content. For lower C contents, the strong enrichment of S causes solute drag pressure, reducing the effective GB mobility. At higher C contents, C replaces S at the GB and thus significantly decreases the solute drag pressure. As a result, austenite grain growth accelerates with higher carbon contents.