The existence of intra-class spectral variability caused by differential scene components and illumination conditions limits the improvement of endmember extraction accuracy, as most endmember extraction algorithms directly find pixels in the hyperspectral image as endmembers. This paper develops a quadratic clustering-based simplex volume maximization (CSVM) approach to effectively alleviate spectral variability and extract endmembers. CSVM first adopts spatial clustering based on simple linear iterative clustering to obtain a set of homogeneous partitions and uses spectral purity analysis to choose pure pixels. The average of the chosen pixels in each partition is taken as a representative endmember, which reduces the effect of local-scope spectral variability. Then an improved spectral clustering based on k-means is implemented to merge homologous representative endmembers to further reduce the effect of large-scope spectral variability, and final endmember collection is determined by the simplex with maximum volume. Experimental results show that CSVM reduces the average spectral angle distance on Samson, Jasper Ridge and Cuprite datasets to below 0.02, 0.06 and 0.09, respectively, provides the root mean square errors of abundance maps on Samson and Jasper Ridge datasets below 0.25 and 0.10, and exhibits good noise robustness. By contrast, CSVM provides better results than other state-of-the-art algorithms.