This article introduces the concept of the driver-identified supervisory control system, which forms a novel architecture of adaptive energy management for hybrid electric vehicles (HEVs). As a man-machine system, the proposed system can accurately identify the human driver from natural operating signals and provides driver-identified globally optimal control policies as opposed to mere control actions. To help improve the identifiability and efficiency of this control system, the method of spectrum-guided fuzzy feature extraction (SFFE) is developed. First, the configuration of the HEV model and its control system are analyzed. Second, design procedures of the SFFE algorithm are set out to extract 15 groups of features from primitive operating signals. Third, long-term and short-term memory networks are developed as a driver recognizer and tested by the features. The driver identity maps to corresponding control policies optimized by dynamic programming. Finally, the comparative study includes involved extraction methods and their identification system performance as well as their application to HEV systems. The results demonstrate that with help of the SFFE, the driver recognizer improves identifiability by at least 10% compared to that obtained using other involved extraction methods. The improved HEV system is a significant advance over the 5.53% reduction on fuel consumption obtained by the fuzzy-logic-based system.