Fast solar wind (SW) flows outward from polar coronal holes (PCHs). The latitudinal extent of the fast SW varies during different phases of the solar cycle. The fast SW in the inner heliosheath produces a flatter proton spectrum than the slow SW that can be observed through energetic neutral atoms (ENAs) by the Interstellar Boundary Explorer (IBEX). In this study, we investigate the evolution of PCHs as reflected in the high-time resolution ENA flux measurements from IBEX-Hi, where the PCHs are identified by ENA spectral indices <1.8. The ENA spectral index over the poles shows a periodic evolution over the solar cycle 24. The surface area with flatter ENA spectra (<1.8) around the ecliptic south pole increases slightly from 2009–2011 and then decreased gradually from 2012–2014. The PCH completely disappears in 2016 and then starts to appear again starting in 2017, gradually growing until 2019. This evolution shows a clear correlation with the change in the PCH area observed at the Sun once the delay in the ENA observation time is included. In addition, the higher-cadence ENA data at the highest latitudes show a rapid evolution of the ENA spectrum near the south pole in 2014 and 2017. The rapid evolution in 2014 is related to a rapid closing of PCHs in 2012 and that in 2017 is related to a rapid opening of PCHs in late 2014. These results also agree qualitatively with the evolution of the ENA spectral index from simulations using a simple time-dependent heliospheric flow model.