There has been a resurgence of interests in stimulated Brillouin scattering (SBS) in optical fibers recently. This is largely due to the need to overcome SBS for power scaling of single frequency fiber lasers. Complex acoustic waveguide designs have been proposed for SBS suppression in optical fibers. There is, therefore, a strong need for finding acoustic modes in complex acoustic waveguides. Furthermore, leaky acoustic modes are often ignored in recent works on SBS in optical fiber. Many leaky acoustic modes involved in SBS in optical fibers often have comparable losses to guided acoustic modes. The losses of both guided acoustic modes and many leaky acoustic modes are dominated by the extremely high material loss of acoustic waves in the GHz region in optical fibers. Therefore, it is very important to consider these leaky acoustic modes in SBS in optical fibers, especially for acoustic antiguide designs used for SBS suppressions, where those leaky acoustic modes are often responsible for the peak SBS gain and consequently SBS threshold. Even for optical fibers with guided acoustic waveguide, fine features in SBS gain spectra can be better understood by studying the additional contributions from leaky acoustic modes. In this work, we report, for the first time, a complex vector acoustic mode solver capable of finding both guided and leaky acoustic modes in arbitrary circular acoustic waveguides. The validity of the mode solver is verified by comparing simulated SBS gain spectra in two optical fibers, one with acoustic guide and one with acoustic antiguide, to measured ones. This acoustic mode solver can be used to provide highly accurate SBS gain spectrum in optical fibers. It will be a critical tool in optical fiber designs for SBS suppressions and enhancements. It will also be very important tool for understanding fine spectral details of SBS in optical fibers.