Achieving carbon neutrality by 2050 has become a global goal, sparking concerns regarding energy consumption and carbon emissions in building operations. Office buildings in high-rise high-density cities serve as central business districts, contributing significantly to the city’s economic activity and consuming a lot of energy. The process of retrofitting existing office buildings for energy efficiency in high-rise high-density cities tends to be challenging. However, there is a lack of comprehensive understanding of the challenges involved in office buildings’ energy retrofitting, as they have not been thoroughly explored. This study aims to investigate the challenges to the existing office building energy retrofitting (EOBER) in high-rise high-density cities with real cases in Hong Kong. Initially, a systematic literature review was undertaken to identify 49 potential EOBER challenges and categorized into seven groups: technical, financial, institutional, social, environmental, regulatory, and other categories. Afterward, 23 EOBER challenges were identified through 24 semi-structured interviews with 36 real office building energy retrofitting cases in Hong Kong. Moreover, these challenges were quantified by the Z-numbers-based Delphi survey and analysis. Results show that regulatory challenges are the primary challenges, followed by financial challenges. The lack of government incentives, policies, legislation and regulations significantly hinders practitioners’ ability to engage in energy retrofitting initiatives. The long payback period of building energy retrofitting poses a critical financial concern for practitioners embracing such initiatives. In the end, this research proposed integrated strategies to tackle these challenges and increase building energy efficiency, including launching financial and regulatory incentives, shortening the interval for mandatory energy audits, disseminating knowledge, and diversifying finance channels of building energy retrofitting. The findings contribute to the body of knowledge by employing systems thinking to identify and evaluate EOBER challenges in high-rise high-density cities through empirical methodologies. Moreover, this study provides valuable references for practitioners in navigating these challenges and minimizing risks associated with the retrofitting process.