The aim of the present work is to develop a computational model of a compact pressurized water nuclear reactor (PWR) to investigate the use of innovative materials to enhance the biological shielding effectiveness. Two radiation transport codes were used: the first one – MCNP – for the PWR design and the GEM/EVENT to simulate (in a 1D slab) the behavior of several materials and shielding thickness on gamma and neutron radiation. Additionally MATLAB Optimization Toolbox was used to provide new geometric configurations of the slab aiming at reducing the volume and weight of the walls by means of a cost/objective function. It is demonstrated in the reactor model that the dose rate outside biological shielding has been reduced by one order of magnitude for the optimized model compared with the initial configuration. Volume and weight of the shielding walls were also reduced. The results indicated that one-dimensional deterministic code to reach an optimized geometry and test materials, combined with a three-dimensional model of a compact nuclear reactor in a stochastic code, is a fast and efficient procedure to test shielding performance and optimization before the experimental assessment. A major outcome of this research is that composite materials (ECOMASS 2150TU96) may replace (with advantages) traditional shielding materials without jeopardizing the nuclear power plant safety assurance.