Research on energy accumulation and releasing in the rock plays a key role on revealing its failure mechanism. This paper establishes a microscopic structure model of granite using Otsu digital image processing (DIP) technology and particle flow code software (PFC2D). A series of numerical compression tests under different confining pressures were conducted to investigate the macro and micro characteristics of energy evolution in granite. The results showed that the energy evolution of granite is divided into three stages: stable accumulation, slow dissipation, and rapid release. With increasing confining pressure, the strain energy accumulation ratio decreased exponentially and the peak value of strain energy increased linearly. It was found that the energy accumulation speed in the pre-peak stage increased as a linear function, while the energy release speed in the post-peak stage decreased as an exponential function. In addition, the feldspar is the main microstructure which played a major part in accumulating energy in granite. However, the unit mineral energy of mica particles was bigger than that of feldspar and quartz. When subjected to increasing confining pressure, the feldspar’s total energy growth rate was fastest. Meanwhile, the mica’s unit energy growth rate was fastest.