This study aims to develop a novel vibratory compaction instrument designed for the preparation of large triaxial soil specimens, offering an efficient alternative to the traditional manual specimen preparation process. However, when the motor frequency closely aligns with the natural frequency of the structure, severe resonance phenomena may occur. This paper employed finite element modal analysis, experimental modal analysis, and operational modal analysis techniques to conduct a multi-objective optimization of the structure, aiming to improve the dynamic characteristics. First, the natural frequency and mode shapes of the vibratory compaction instrument were determined via finite element technology. Subsequently, the optimization of this instrument was conducted by integrating the Kriging surrogate model with the Non-dominated Sorting Genetic Algorithm II (NSGA-II). Finally, the steel column underwent a modal test using impulse stimulation to ascertain its natural frequency, and the stochastic subspace identification (SSI) approach was employed to conduct an operational modal analysis of the vibratory compaction instrument to determine its modal characteristics. The experimental findings corroborated the conclusions obtained from the finite element analysis, validating the precision of the finite element modal analysis and the rationality of the optimized structure. The research results are of significant importance for improving the preparation efficiency of large triaxial specimen, contributing to advancements in soil mechanics testing.