The utilization of metal/semiconductor Schottky devices for plasmonic harvesting of hot carriers holds immense potential in the field of sub-bandgap photodetection. In this work, we explore a surface passivation scheme using air plasma exposure to modify the Si (100) surface and subsequently the crystal orientation of the deposited Ag film for photon detection in the mid-infrared (MIR) regime. This tailoring was achieved by varying the plasma exposure duration (0, 150, 300, 450, and 600 s). As a result, we could tune the crystal orientation of Ag from the (200) to the (210) crystal plane, with the Ag (111) orientation present in all devices. Furthermore, the photo-response behavior under MIR exposure at λ = 4.2 µm was studied both experimentally and using COMSOL simulations. It was observed that both photoelectric (PE) and photothermal (PT) effects contributed to the photo-response behavior of all devices. The Ag/Si device exposed to air plasma for 300 s exhibited the maximum PE-driven response (2.73 µA/W), while the device exposed to air plasma for 600 s showed a significant PT-driven response (13.01 µA/W). In addition, this strategy helped reduce the reverse leakage current by up to 99.5%. This study demonstrates that MIR detection at longer wavelengths can be optimized by tailoring the crystal orientation of the metal film.