The identification of structural damage with the unavailability of input excitations is highly desired but challenging since structural dynamic responses are affected by the coupling effect of structural parameters and external excitations. To deal with this issue, in this paper, an output-only damage identification strategy based on swarm intelligence algorithms and correlation functions of strain responses is proposed to identify structures subjected to single or multiple unknown white noise excitations. In the proposed strategy, four different population-based optimization algorithms—particle swarm optimization, the butterfly optimization algorithm, the tree seed algorithm, and a micro search Jaya (MS-Jaya)—are employed and compared. The micro search mechanism is integrated into a basic Jaya algorithm to improve its computational efficiency and accuracy by eliminating some damage variables with small values for the identified best solution after several iterations. The objective function is established based on the proposed auto/cross-correlation function of strain responses and a penalty function. The effectiveness of the proposed method is verified with numerical studies on a simply supported beam structure and a steel grid benchmark structure under ambient excitation. In addition, the effect of the reference point, number of sensors, and arrangement of strain gauges on the performance of the proposed method are discussed in detail. The investigated results demonstrate that the proposed approach can accurately detect, locate, and quantify structural damage with limited sensors and 20% noise-polluted strain responses. In particular, the proposed MS-Jaya algorithm presents a more superior capacity in solving the optimization-based damage identification problem than the other three algorithms.