This work presents a fully automated machine vision system that detects and dimensions color marked defects on steel billets to guide an industrial robot, which then grinds out these defects. There are three main modes of operation: 1) Real-time color detection: A color camera observes the surface of the steel bar during its transport, with the aim of detecting the colored patches that mark a defect to be processed. Thereby, the sequencing of several color image processing algorithms yields a logical sequence of pipelined decision algorithms that are used to reliably determine bounding boxes delimiting the regions to be ground. 2) 3D modeling: Following the detection of a color marking, the geometry of the bar is measured using two plane-of-light scanners. The geometric measurements combined with the bounding box from the color detection deliver the necessary information to construct a 3D model for that portion of the bar, which needs to be ground. In the process, metric coordinates are obtained by means of projective geometry and algebraic as well as geometric least squares fitting techniques are applied to generate the 3D model. 3) Calibration: A calibration procedure is used to individually calibrate the coordinates of the machine vision system and to register them with the coordinate system of the robot. A direct linear transformation algorithm is used to determine an initial estimate for a nonlinear iterative Gauss-Newton optimization of the calibration parameters.