Nowadays, tool condition monitoring (TCM), which can prevent the waste of resources and improve efficiency in the process of machining parts, has developed many mature methods. However, TCM during the production of cutting tools is less studied and has different properties. The scale of the defects in the tool production process is tiny, generally between 10 μm and 100 μm for diamond tools. There are also very few samples with defects produced by the diamond tool grinding process, with only about 600 pictures. Among the many TCM methods, the direct inspection method using machine vision has the advantage of obtaining diamond tool information on-machine at a low cost and with high efficiency, and the method is accurate enough to meet the requirements of this task. Considering the specific, above problems, to analyze the images acquired by the vision system, a neural network model that is suitable for defect detection in diamond tool grinding is proposed, which is named DToolnet. DToolnet is developed by extracting and learning from the small-sample diamond tool features to intuitively and quickly detect defects in their production. The improvement of the feature extraction network, the optimization of the target recognition network, and the adjustment of the parameters during the network training process are performed in DToolnet. The imaging system and related mechanical structures for TCM are also constructed. A series of validation experiments is carried out and the experiment results show that DToolnet can achieve an 89.3 average precision (AP) for the detection of diamond tool defects, which significantly outperforms other classical network models. Lastly, the DToolnet parameters are optimized, improving the accuracy by 4.7%. This research work offers a very feasible and valuable way to achieve TCM in the manufacturing process.