Ring core fibers (RCFs) offer unique advantages in fiber image transmission, as their weakly-coupled orbital angular momentum mode groups result in high resolution images. However, severe image distortion is still exhibited during fiber transmission when subjected to strong disturbances. Here, we present a novel approach with a differential neural network, namely the polarization speckle differential imaging (PSDI) method, to significantly enhance both the robustness and image resolution of RCF-based imaging systems. When the fibers are disturbed, the PSDI method establishes the perturbation mapping relationship between two polarization speckles and utilizes a differential method to eliminate the perturbation effect in the speckles. This approach mitigates environmental disturbances, resulting in an enhancement of the imaging system's robustness in dynamic environments. By addressing the limitations of conventional techniques, this research represents a significant advancement in the field of fiber imaging technology, with potential applications ranging from endoscopy to high-resolution imaging in complex and dynamic settings.