In a realistic scenario where a large number of workpieces need to be measured, any measurement method that can detect roughness only for a single workpiece is very limited in terms of measurement efficiency. To address this problem, a multi-object surface roughness detection model based on Faster R-CNN is proposed in this paper. The model features milled workpiece images with a convolutional neural network. And the obtained features will feed into a Region Proposal Network for inferring those regions where workpieces may be present. The regions and features go through a ROI pooling layer and a predictor to get more accurate target regions and measure the roughness of the workpieces in the regions. The experimental results show that the model proposed in this paper can accurately detect those regions where workpieces are present in the image and detect the corresponding roughness grade of the workpieces. A mean average precision of 97.80% and a detection speed of 5.82 fps for the test set of milled workpieces were achieved by the model under different placement angles and variable light conditions.