Strong interaction between circularly polarized light and chiral plasmonic nanostructures can enable controllable asymmetric photophysical processes, such as selective chiral switching of a plasmonic nanorod-dimer. Here, we uncover the underlying physics that governs this chiral switching by theoretically investigating the interplay between asymmetric photothermal and optomechanical effects. We find that the photothermally induced local temperature rises could play a key role in activating the dynamic chiral configurations of a plasmonic dimer due to the temperature-sensitive molecular linkages located at the gap region. Importantly, different temperature rises caused by the opposite handedness of light could facilitate selective chiral switching of the plasmonic dimer driven by asymmetric optical torques. Our analyses on the wavelength-dependent selectively chiral switching behaviors are in good agreement with the experimental observations. This work contributes to a comprehensive understanding of the physical mechanism involved in the experimentally designed photoresponsive plasmonic nanosystems for practical applications.