Spectral mixing is frequently encountered in remotely sensed images while imaging heterogeneous surfaces. Spectral mixing occurs at the macroscopic level due to the sensor’s low spatial resolution or multiple reflections from materials received by the sensor. Due to the mixing effect, retrieving the information of spectral endmembers becomes complicated. In this study, two different unmixing methods: (i) linear mixing models (LMM) and (ii) the Hapke nonlinear mixing model, were used to quantitatively estimate the fractional abundance of various rock types that will be helpful in remotely mapping the mineral prospects. Three types of constraints using the least square approach were applied independently to LMM and Hapke models, namely FCLS (sum to unity and nonnegativity constrained), NNLS (nonnegativity constrained), and UCLS (unconstrained). AVIRIS-NG (Airborne Visible and InfraRed Imaging Spectrometer-Next Generation) and ASTER (Advanced Spaceborne Thermal Emission and Reflection Radiometer) images from the Sittampundi anorthosite complex (SAC) were utilised for the spectral mixture analysis. Few studies are available on the anorthosite rocks of Sittampundi as an analogue to the Lunar anorthosite. The study area is known for its chromite deposits occurring in pyroxenite rocks. Therefore, in the present study, endmembers considered are anorthosite and pyroxenite/metagabbro.The moderately dense vegetation cover of the study area obscures the detection of rock outcrops on the surface. The study area exhibits nonlinear spectral mixing effects due to the vegetation cover. Hence apart from the two rock endmembers, vegetation is also considered as an endmembers in spectral mixture analysis. The results show a high fractional abundance of vegetation and anorthosite throughout the study region, with the pyroxenite/metagabbro mainly concentrated over the western and eastern regions. The Hapke model shows more accurate results than LMM on the AVIRIS-NG image. The Hapke model outperforms LMM in accurately retrieving the fractional abundance of endmembers due to the nonlinear spectral mixing effects caused by vegetation. Additionally, the constrained least squares methods, i.e., FCLS and NNLS, performed better than UCLS in linear and nonlinear mixing models, resulting in accurate and reliable abundance maps. AVIRIS-NG outperformed the ASTER images in fractional abundance mapping of endmembers, highlighting the importance of utilising higher spatial and spectral resolution data for more reliable land cover and lithology mapping. Overall, this study demonstrates the potential of spectral mixture analysis in identifying geologic outcrops in vegetation-covered regions. Hence, the methodology presented in this study benefits the lithological mapping of vegetation-covered regions and has wide applications in quantitatively assessing the lithology of mineral prospects.