Differently heat-treated rail materials develop characteristic hardness profiles near the rail surface. Plastic deformation due to contact loading is dependent on these material properties. The deformation causes crack initiation near the surface. The optimization and development of rail materials requires the quantification of these deformations below and close to the contact area. An approach to assess such wheel-rail contacts is the finite element method. The main difficulty is the necessary multi-scale approach needing very fine elements to account for the micro and macro behavior near the contact. For this purpose, within the present work a FE modelling method is developed to simulate the plastic deformation in laboratory tests while accounting for the influence of the surface roughness of the test specimens. In detail, a methodology is developed to correlate the plastic deformation to the contact load in the elements near the surface and extrapolate this deformation to obtain the deformation gradient in the outermost element which is affected by the roughness of the surface. This calculated deformation gradient is calibrated by the measured deformation gradient from laboratory tests and for differently heat-treated rail materials. The developed method is applicable to rolling sliding contact simulations using the commercial FE code ABAQUS.