Highwall mining is a method that involves using a continuous highwall miner system (CHM) to extract coal from the remaining coal seams, which has proven to be an effective and safe method for extracting coal after open-pit mining. However, application cases globally have shown that the feasibility of highwall mining in open-pit coal mines is subject to geological conditions, mining techniques, and other factors. If application conditions are not suitable, equipment may be trapped under collapsed coal–rock masses and unable to be retrieved, resulting in severe safety issues for slope stability. To meet the real-world demand for extracting the remaining coal in open-pit coal mines in China, it is urgent to conduct a feasibility evaluation of highwall mining in these areas. This paper establishes a mathematical evaluation framework for assessing the feasibility of highwall mining by summarizing a large number of engineering application cases globally and analyzing various technical characteristics such as geological deposit conditions, mining techniques, and technical equipment. The analytic hierarchy process (AHP), fuzzy comprehensive evaluation (FCE) and variable weight theory (VWT) are utilized in conjunction to form this framework, which includes four secondary indicators: geological deposit factors, mining technique factors, safety impact factors, and economic evaluation factors, and 20 tertiary sub-indicators, along with their corresponding characteristic values. The feasibility sub-set is divided into four categories: infeasible, basically feasible, relatively feasible, and highly feasible, and the values of the sub-indicators strictly follow and represent these four levels of feasibility. Weight vectors for the sub-indicators are obtained through a judgment matrix established within the mathematical evaluation framework. The fuzzy relationship matrix of the sub-indicators is constructed using fuzzy mathematical membership functions, and the final feasibility evaluation is determined through two-level comprehensive evaluation. The accuracy of the model is verified using the characteristic parameters of open-pit coal mines under two different conditions (JZT coal mine in Inner Mongolia, China, and GC coal mine in Australia). The results demonstrate that the maximum evaluation membership degree for the JZT mine is 0.7113, belonging to the “highly feasible” level, while the GC mine is 0.3304, belonging to the “basically feasible” level, which aligns well with real-world usage, proving that the evaluation model can effectively reveal the performance and membership degree of each indicator in different application cases. By quantitatively characterizing the feasibility level of highwall mining technology under different application conditions, this evaluation model can provide scientific guidance for coal mining enterprises to introduce CHM for highwall mining operation in open-pit coal mines.