The college building is a large energy consumer with a high density of energy consumption. However, less attention is paid to college buildings, particularly college dormitory buildings. Based on the one-year historical data collected from 20 college dormitory buildings located in Wuhan, China, this study aims to propose a three-stage strategy to identify and analyze the energy consumption patterns and characteristics of college dormitories in detail, including determining energy consumption patterns, analyzing key characteristics based on four indexes, and examining three influencing factors (occupants’ gender and floor and orientation location of rooms). The results show that the heavy energy users (around 10% of all occupants) consume around 20% of the total energy and have the narrowest comfort temperature range. However, the light energy users, 42% of total occupants, consume only approximately 27% of total energy. Their different tolerance to coldness is the main reason contributing to different energy consumption. The dormitories of males and location of the top floor and corner tend to consume significantly more energy in hot weather. This study would help campus facilities to understand the energy use behavior of occupants and formulate adequate policies so as to improve the energy management of campuses.