Composites of polymer and micron-sized particles of carbonyl-iron were investigated in terms of their magnetization behavior. Thermoplastic elastomers with varying Young's modulus (EPolymer=0.14–14.6 MPa) were used as matrix material. Field dependent magnetization curves reveal that the hysteretic behavior of the composites strongly depends on both the particle fraction (7, 10, 14, 21, 31 vol%) and on the mechanical properties of the polymer. It is shown that hysteresis only appears above a certain fraction of magnetic particles which can be accounted to the magnetic exchange between the particles. However, hysteresis is suppressed in the composite with largest Young's modulus of the polymer matrix, even at largest particle fraction.