This paper studied an unsupervised machine learning method to achieve real-time diagnosis of tip wear in tip-based nanomachining with an Atomic Force Microscope (AFM). The unsupervised Gaussian Mixture Model (GMM) was applied for online pattern recognition using the collected process data. In this work, the time varying signal of machining force was collected and managed in the form of moving data windows. The characteristics of the nanomachining force signal, including the maximum force, peak to peak force, and the variance, were calculated as the data of extracted features to indicate the tip wear conditions. Outliers were identified and removed using the Mahalanobis Distance Detection. A GMM model was trained using the historical processing data, in which the tip wear states were identified by clustering the feature parameters of each data frame. The change of the tip conditions was detected and tracked through the changing numbers of the tip failure points in the data windows. Experiment results showed that the highest and the average recognition accuracy of the proposed model was 1 and 0.8367 with the longest and the average calculation time of 0.031 s and 0.025 s. Compared with supervised and semi-supervised machine learning algorithms, the GMM based on unsupervised machine learning model has slightly lower recognition accuracy, but much shorter calculation time, which is suitable for online automatic diagnosis of tool wear in large-scale nanomanufacturing in real time.