In recent years, growing attention has been paid to the buffer layer ablation failure in high voltage cables. The buffer layer was always regarded as a homogeneous material in previous studies, while in this paper, the effects of its structural inhomogeneity on failure is found unneglectable. The buffer layer is recognized to be a porous material comprising fibers and air pores through microscopic characterization. Its porosity is measured to be 85% by testing mercury intrusion porosimetry. Especially, some small fiber protrusions are observed on the surface of buffer layer, which are categorized and named as tip protrusions and arched protrusions. Subsequently, the influence of these fiber protrusions is investigated by quantitatively proposing their geometric models. The electric field distortion is observed at the fiber regions after finite element simulation. Under the extreme circumstances when the tip protrusions are vertical and close to the central area of air gap, the maximum electric field at tip site can reach to 28.5 kV/mm, which is 9.5 times of the breakdown threshold of air. And the maximum electric field of arched protrusions region can reach to 5.94 kV/mm, which is more moderate but still higher than the threshold. Therefore, it is demonstrated that the breakdown of air gap inner cables can be induced by the inhomogeneous fiber protrusions.