Ballast degradation can lead to adverse effects such as inadequate drainage, track settlement and reduced lateral stability, which could compromise track safety, daily functionality, and long-term maintenance. Field inspection of ballast for monitoring degradation and functional performance is a challenging task. Current state-of-the-practice methods for evaluating ballast primarily depend on subjective visual inspection, labor-intensive sampling, laboratory sieve analyses or Ground Penetrating Radar (GPR) technology. These methods fall short in providing an in-depth assessment of ballast, specifically in determining the degradation level and aggregate size and shape characteristics at various depths. In this regard, this research developed an innovative ballast investigation platform, the Ballast Scanning Vehicle (BSV), to automate the processes of acquiring detailed ballast inspection data. The BSV utilizes a deep learning-based pipeline for image segmentation to evaluate task-specific metrics such as coarse aggregate gradation, Fouling Index (FI), and continuous track FI depth profiles. This paper provides a detailed overview of the BSV’s functions as well as the different modules of the deep learning-based pipeline. Validation of the BSV’s capabilities was conducted at the Transportation Technology Center (TTC) and is discussed in detail. Based on the field results, the BSV is capable of providing accurate and near real-time evaluation of in-service ballast conditions, serves as a robust means for inspecting long sections of track, and can be used to investigate persistent trouble-spots related to track performance.