The solute drag effect of phosphorus (P) in the single-phase austenite (γ-Fe) region was studied under isothermal annealing conditions at temperatures of 1050 °C, 1150 °C, 1250 °C and 1350 °C. High-temperature laser scanning confocal microscopy (HT-LSCM) was used to observe and quantify in situ grain growth on the surface of three different samples containing 0.026 wt.-% P, 0.044 wt.-% P and 0.102 wt.-% P. The dependence of the arithmetic mean grain sizes on time and temperature were modeled mathematically using a simple ordinary differential equation (ODE) according to normal grain growth (NGG) theory. As no other major effects, i.e., Zener-pinning by precipitates, occurred under the selected experimental conditions, grain growth interference was only considered by grain boundary (GB) segregation of P. Thus, the total grain boundary mobility (M) was directly determined depending on the P content and isothermal annealing temperature. The fitted GB mobility values enabled the determination of an average binding energy value between impurity P atoms and grain boundaries (E0 = -1.1 - -0.6 eV) in the system. Finally, the GB segregation of P in γ-Fe was derived from the observed grain growth kinetics. The results showed reasonable agreement with calculations using segregation enthalpies from the literature.