Semiconductor wafer is widely used in welding robot, spray robot, unmanned material delivery vehicle, and detection station sensor. The defects of semiconductor wafer, such as stains, burrs, scratches, and holes generated in the manufacturing process, severely affect the quality of downstream products. Therefore, the inspection of wafer defect could not be neglected. Traditional semiconductor wafer defect inspection methods based on handcrafted features heavily rely on the expertise and are limited in some application scenario. In this article, a novel method based on deep convolutional neural networks for semiconductor wafer surface defect inspection is proposed. First, a new structure of feature pyramid networks with atrous convolution (FPNAC) is developed to extract the features and to generate feature maps. Second, the feature maps are fed into region proposal network (RPN) to generate region proposals. Finally, the region proposals are aligned to corresponding size as the inputs of deep multibranches neural network (DMBNN) consisting of three branches, to classify and segment the defects precisely. Experimental results demonstrate that the proposed method yields good comprehensive performance with mean pixel accuracy (MPA) 93.97% and mean intersection over union (MIoU) 83.58%.