High-resolution and thermal infrared hyperspectral data acquired from the Thermal Infrared Airborne Spectrographic Imager (TASI) have been recognized as efficient tools in geology, demonstrating significant potential for rock discernment. Deep learning (DL), as an advanced technology, has driven substantial advancements in lithological mapping by automatically extracting high-level semantic features from images to enhance recognition accuracy. However, gathering sufficient high-quality lithological samples for model training is challenging in many scenarios, posing limitations for data-driven DL approaches. Moreover, existing sample collection approaches are plagued by limited verifiability, subjective bias, and variation in the spectra of the same class at different locations. To tackle these challenges, a novel sample generation method called multi-lithology spectra sample selection (MLS3) is first employed. This method involves multiple steps: multiple spectra extraction, spectra combination and optimization, lithological type identification, and sample selection. In this study, the TASI hyperspectral data collected from the Liuyuan area in Gansu Province, China, were used as experimental data. Samples generated based on MLS3 were fed into five typical DL models, including two-dimensional convolutional neural network (2D-CNN), hybrid spectral CNN (HybridSN), multiscale residual network (MSRN), spectral-spatial residual network (SSRN), and spectral partitioning residual network (SPRN) for lithological mapping. Among these models, the accuracy of the SPRN reaches 84.03%, outperforming the other algorithms. Furthermore, MLS3 demonstrates superior performance, achieving an overall accuracy of 2.25–6.96% higher than other sample collection methods when SPRN is used as the DL framework. In general, MLS3 enables both the quantity and quality of samples, providing inspiration for the application of DL to hyperspectral lithological mapping.