Ultrasound elasticity imaging techniques aim to provide a non-invasive characterization of tissue mechanical properties to detect pathological changes and monitor disease progression. Harmonic motion imaging (HMI) is an ultrasound-based elasticity imaging technique that utilizes an oscillatory acoustic radiation force to induce localized displacements and estimate relative tissue stiffness. Previous studies have applied a low amplitude modulation (AM) frequency of 25 or 50 Hz in HMI to assess the mechanical properties of different tissue types. In this study, we investigate the dependence of AM frequency in HMI and whether the frequency can be adjusted based on the size and mechanical properties of the underlying medium for enhanced image contrast and inclusion detection. A tissue-mimicking phantom with embedded inclusions at different sizes and stiffnesses was imaged within a range of AM frequencies from 25 to 250 Hz at 25-Hz step size. The AM frequency at which the maximum contrast and CNR are achieved depends on the size and stiffness of the inclusions. A general trend shows that contrast and CNR peak at higher frequencies for smaller inclusions. In addition, for some inclusions with the same size but different stiffnesses, the optimized AM frequency increases with the stiffness of the inclusion. Nevertheless, there is a shift between the frequencies at which the contrast peaks and those with maximum CNR. Finally, in agreement with the phantom findings, imaging an ex-vivo human specimen with a 2.7-cm breast tumor at a range of AM frequencies showed that the highest contrast and CNR are achieved at the AM frequency of 50 Hz. These findings indicate that the AM frequency can be optimized in different applications of HMI, especially in the clinic, for improved detection and characterization of tumors with different geometries and mechanical properties.