The fully microscopic $p$-nucleus optical potential has been calculated in the framework of the first order Brueckner theory employing Urbana V14, soft-core internucleon interaction along with the relativistic mean field densities both for protons and neutrons. It is observed that the volume integral per nucleon, of the real part of the spin-orbit interaction calculated for Zr ($A=76\text{\ensuremath{-}}110$) and Sn ($A=96\text{\ensuremath{-}}136$) isotopes, decreases with the increase in neutron number. The present optical model calculation satisfactorily reproduces the experimental (where available) cross sections and analyzing power. Further the magnitude of the first maximum (minimum) in the calculated analyzing power decreases (increases) with the addition of neutrons both for Zr and Sn isotopes reflecting the weakening of the spin-orbit interaction.