In the context of the European Critical Raw Materials Act, this work attempts to demonstrate the potential of residual material flows from non-ferrous metallurgy and their possible contribution to the supply security of metals by locally available new secondary resources, assuming technically and economically viable processing. Based on the aluminium, zinc, copper and lead industries, the resulting waste streams are discussed and, in particular, the complex process consisting of physical, chemical and metallurgical steps is described. Their diversity, be it slags, dusts or even sludges, has a wide variety of morphologies and compositions due to the process of generation. In the past, many concepts for reprocessing were investigated, but the goal was usually only the recovery of one target element or to avoid landfilling by using it, for example, as a building material, whereby the metals contained are completely lost. If the target is the extraction of valuables, the required interdisciplinary process development must be based on an in-depth characterization to understand the behaviour of metals and trace elements in possible extraction steps and also to develop suitable strategies for influencing the behaviour of target elements with the aim of extraction. This starts with an in-depth comprehension of the formation process, which is the subject of this article and has a direct influence on the composition and morphology of the materials, thus forming the basis for understanding the behaviour in potential recycling processes. Furthermore, typical compositions of the residual material streams, sources and, if available, quantities are shown and, in summary, an attempt is made to evaluate the materials in a SWOT analysis and to address the challenges in developing extraction steps for processing. While mine tailings are mostly found outside of Europe, the potential of the residual materials from metallurgy is local due to the processing of the concentrates in Europe. This leads to several potential advantages in a possible reprocessing, such as no or shorter transport routes, which is linked to lower quantity of emissions, defined volume and known composition, no geopolitical risk, conservation of primary resources, and increasing Europe’s sustainability through a more comprehensive use of the raw materials.