Engineering of the cross-section shape and size of ultra-scaled Si nanowires (SiNWs) provides an attractive way for tuning their structural properties. The acoustic and optical phonon shifts of the free-standing circular, hexagonal, square, and triangular SiNWs are calculated using a modified valence force field (MVFF) model. The acoustic phonon blue shift (acoustic hardening) and the optical phonon red shift (optical softening) show a strong dependence on the cross-section shape and size of the SiNWs. The triangular SiNWs have the least structural symmetry as revealed by the splitting of the degenerate flexural phonon modes and these show the minimum acoustic hardening and the maximum optical hardening. The acoustic hardening, in all SiNWs, is attributed to the decreasing difference in the vibrational energy distribution between the inner and the surface atoms with decreasing cross-section size. The optical softening is attributed to the reduced phonon group velocity and the localization of the vibrational energy density on the inner atoms. While the acoustic phonon shift shows a strong wire orientation dependence, the optical phonon softening is independent of wire orientation.