We have numerically investigated the influences of asymmetrical geometric structures on the birefringence of the index-guiding photonic crystal fiber (IG-PCF), by the finite element method. The structure of a reported highly birefringent (HB) PCF with artificial defect in the core is modified to destruct the symmetry. Our proposed structural modifications include elliptical air hole cladding, squeezing lattice of air holes, rotational elliptical air holes, and horizontally shifted air holes, in the cladding. It is found that the birefringence is sensitive to the modifications in different cases due to the different mechanisms. For 1550 nm wavelength, the optimized birefringence can be obtained to be 0.0328 by the elliptical air holes with the major semi-axis in the direction of the long side of the rectangular core region. Furthermore, for the HB-PCF by the asymmetric core shape, the structural factor restricting the further increase of birefringence is that continuous increase of asymmetry leading to the multi-core transmission. It seems to be that This work is not only a proposing of the optimal design of HB-PCF, but also a reference of founding the essence of geometric birefringence.