The influence of grain growth inhibition by pearlite on hydrogen embrittlement (HE) behavior of ultra-low carbon ferritic steels was studied. The Fe-0.02C alloy has a considerable sensitivity to HE, while the Fe and Fe-0.1C materials have lower HE sensitivity. The high fraction of high-angle grain boundaries (HAGB) is responsible for the shift of the peak to a higher temperature in the Fe-0.1C alloy. The trapping of hydrogen by HAGBs was observed by hydrogen mapping. Higher density of HAGBs contributes to higher trap density in Fe-0.1C alloy and lower H diffusion coefficient. The coexistence of hydrogen enhanced decohesion (HEDE) and hydrogen-enhanced localized plasticity (HELP) mechanisms was identified and discussed. The results indicate that HEDE is active in the initial stage of the tensile loading during crack initiation and HELP in the rest of tensile testing duration that controls the crack propagation. Due to the reduction in grain size caused by the addition of pearlite, the normalized hydrogen content per HAGB length is significantly lower than Fe-0.1C, which means that the critical amount of local hydrogen required for crack initiation is less likely to accumulate due to the weakening of cohesive interatomic strength. In Fe-0.02C alloy, the ciritcal local content was built up at HAGBs, and resulting in activation of HEDE mechanism for crack initiation. In Fe-0.1C, a lower possibility of accumulating the critical H concentration at HAGBs, resulting in limited activation of HEDE-based crack initiation at HAGB and less crack propagation events based on the HELP mechanism.