An efficient and accurate joint neural network has been established to predict the optical properties of different photonic crystal fibers using two-dimensional structural images. The input variables of this network are structural images derived from the two-dimensional material refractive index distribution of the fiber cross-section. The structural images have greater flexibility and facilitate generalization research and transfer learning testing on various types of optical fibers. The generalization properties of this neural network for other photonic crystal fibers with different lattice arrangements have been investigated, and the corresponding transfer learning method is also proposed. With this transfer learning method, only a small amount of data is required to train and update the parameter weights of the last fully connected hidden layers. As a result, efficient and accurate prediction performance can be achieved for other photonic crystal fibers with different lattice arrangements. The proposed joint neural network and the transfer learning method are flexible and efficient, which provide new approaches for more flexible optimization and reverse design of optical fiber structures in the two-dimensional parameter space.