Coarse aggregates play a significant role in configuring the pore structures in the porous asphalt mixture (PAM). This study aims to reveal the influence of coarse aggregates’ morphological characteristics on the pore features in the coarse aggregate skeleton (CAS) of PAM. To achieve this goal, X-ray CT and three-dimensional (3D) visualization reconstruction technology were utilized to acquire the 3D morphology of different kinds of coarse aggregates. Based on the morphology of real aggregate particles, a 3D simulation model of CAS was developed, and virtual scanning imaging technology was used to obtain the pore features in the CAS. The sensitive influence of coarse aggregate morphologies on the pore features was evaluated by GA-BP neural network. Results show that air void in the skeleton is most sensitive to the composite flat-elongated index and composite sphericity of 13.2mm aggregates, and composite sphericity of 9.5mm aggregates. The composite sphericity of 13.2mm aggregates, composite flat-elongated index and composite texture index of 9.5mm aggregates, composite flat-elongated index, composite angularity index, and composite texture index of 4.75mm aggregates are the high-sensitivity influencing factors to the fractal characteristic of pore structures.