Railway sleepers are safety–critical components of a railway structure. They support ballasted track superstructure and are a critical factor in track geometry and track components’ deterioration. Unsupported sleepers are a common issue incurred after tracks have been utilized. When unsupported sleepers are present, they cause differential settlements of track superstructures, additional dynamic loading, and excessive train-track vibrations which affect passenger comfort, safety, and maintenance cost. This study is the world's first to develop new machine learning models to prognose and better diagnose defect severities of unsupported sleepers aligned with practical track inspection guidelines. Data used to develop machine learning models are based on a verified finite element model with actual field measurements, enabling unbiased full data ranges that govern all defect conditions. Different conditions of unsupported sleepers can be explored by varying locations of unsupported sleepers and the number of unsupported sleepers. Also, other operational parameters can be addressed such as speeds of rolling stock, the roughness of rail surface, and vertical stiffness of wheel-rail contact. In total, 2016 data sets have been obtained. Axle box accelerations are adopted as key indicators for machine learning models. Machine learning techniques used in the study are the convolutional neural network, recurrent neural network, ResNet, and fully convolutional neural network. Data fusion and assimilation have been conducted since the data points are dependent on the train speeds. Our new results reveal a breakthrough essential for real-world applications that the convolutional neural network model provides the best accuracy in both unsupported sleeper prognostics and severity identification. The accuracies of detection and severity identification are 99.34% and 97.02% respectively.