Refractory linings are used in steel ladles in iron and steel industry, to protect the vessel structure from the molten steel (with temperature above 1600 °C). To increase the durability of the refractory lining, researching the possible failure causes is of importance. In recent decades, alumina spinel refractories have become a common material in the barrel zone of steel ladles in direct contact with steel. In this regard, the current study employed unit-cell finite element modeling technique to investigate the irreversible material behavior of alumina spinel bricks. At first, a study on the joint size and friction effect was conducted. Then, three distinct constitutive material models were assigned to the working lining, each corresponding to an irreversible deformation mechanism. The Norton-Bailey creep model was used to simulate creep behavior, the Drucker-Prager yield criterion was used to simulate shear failure, and concrete damaged plasticity was used to describe tensile failure. The findings of the three models were compared to understand how each phenomenon affected the lining's and steel shell's stress-strain response. The simulations showed the occurrence location and time of each irreversible behavior. The effect of considering plasticity for the steel shell on the mechanical behavior of the refractory lining was also investigated, which showed a decrease of irreversible strains at the working lining.