Internal tumor motion is commonly predicted using external respiratory signals. However, the internal/external correlation is complex and patient-specific. The purpose of this study was to develop various models based on the radiomic features of computed tomography (CT) images to predict the accuracy of tumor motion tracking using external surrogates and to find accurate and reliable tracking algorithms. Images obtained from a total of 108 and 71 patients pathologically diagnosed with lung and liver cancers, respectively, were examined. Real-time position monitoring motion was fitted to tumor motion, and samples with fitting errors greater than 2 mm were considered positive. Radiomic features were extracted from internal target volumes of average intensity projections, and cross-validation least absolute shrinkage and selection operator (LassoCV) was used to conduct feature selection. Based on the radiomic features, a total of 26 separate models (13 for the lung and 13 for the liver) were trained and tested. Area under the receiver operating characteristic curve (AUC), sensitivity, and specificity were used to assess performance. Relative standard deviation was used to assess stability. Thirty-three and 22 radiomic features were selected for the lung and liver, respectively. For the lung, the AUC varied from 0.848 (decision tree) to 0.941 [support vector classifier (SVC), logistic regression]; sensitivity varied from 0.723 (extreme gradient boosting) to 0.848 [linear support vector classifier (linearSVC)]; specificity varied from 0.834 (gaussian naive bayes) to 0.936 [multilayer perceptron (MLP), wide and deep (W&D)]; and MLP and W&D had better performance and stability than the median. For the liver, the AUC varied from 0.677 [light gradient boosting machine (Light)] to 0.892 (logistic regression); sensitivity varied from 0.717 (W&D) to 0.862 (MLP); specificity varied from 0.566 (Light) to 0.829 (linearSVC); and logistic regression, MLP, and SVC had better performance and stability than the median. Respiratory-sensitive radiomic features extracted from CT images of lung and liver tumors were proved to contain sufficient information to establish an external/internal motion relationship. We developed a rapid and accurate method based on radiomics to classify the accuracy of monitoring a patient's external surface for lung and liver tumor tracking. Several machine learning algorithms-in particular, MLP-demonstrated excellent classification performance and stability.