Urbanization has led to homogenizing heritage site landscapes, and the protective measures have become disconnected from public needs. Additionally, the complex and diverse overall characteristics of heritage sites and the uneven distribution of values across different areas are related to the lack of landscape experience assessment. The “subjective + objective” cognitive evaluation and visual perception framework that adopts the Scenic Beauty Estimation Procedure-Semantic Differential (SBE–SD) method and eye-tracking analysis can compensate for the limitations of a single evaluation method by integrating quantitative and qualitative analysis. This research takes the Yi’an Fortress in Zhangpu County, Fujian Province, as the object and examines the visual experiences of different areas and types of landscapes within the Yi’an Fortress. The findings reveal several key insights: (1) Significant differences were found in the landscape experiences of different areas within the heritage site. The visual experience score of the core building area of Yi’an Fortress is (1.01) > the heritage entrance area (0.897) > the residential area (0.841) > the natural ecological area (0.784), indicating that the natural ecological area should be the focus of future protection and development efforts, with a particular emphasis on enhancing the ‘landscape aesthetic’ and ‘landscape cultural’ aspects. (2) The landscape experience scores can be used to understand the reasons for the differences in participants’ experiences of different landscapes. The architectural heritage landscape of the Yi’an Fortress scored highest in the experience evaluation due to its superior performance in terms of ‘landscape form’ factors. The cultural decorative landscape scored next, while the garden greening landscape scored the lowest, due to their poorer performance in terms of ‘landscape form’ and ‘landscape aesthetics’. (3) The eye-tracking data was consistent with the results of the subjective evaluation, validating the “subjective + objective” cognitive evaluation and visual perception framework, employing the SBE–SD method and eye-tracking analysis is a scientific and effective method for assessing the visual experience of heritage landscapes. These results provide a scientific basis for the heritage planners and managers of the Yi’an fortress to improve the landscape environment, better meet public needs, and preserve the unique character of this vital cultural resource. Furthermore, this study offers a new research method and approach for the protection of other heritage landscapes.