(1) Background: Trace metal (TM) contamination of farmland soil in Taiwan occurs because factories dump wastewater into irrigation ditches. Since vermicompost affects the bioavailability of TMs, the objective of this study was to evaluate its effects on the accumulation of growth of TMs in leafy vegetables. (2) Methods: Two TM-contaminated soils and different types of pak choi and lettuce were used and amended with vermicompost. Besides soil properties, the study assessed vermicompost’s influence on the growth, accumulation, and chemical forms of TMs and on the health risks posed by oral intake. (3) Results: Vermicompost could increase the content of soil organic matter, available phosphorus, exchangeable magnesium, and exchangeable potassium, thus promoting the growth of leafy vegetables. The accumulation of four TMs in crops under vermicompost was reduced compared to the control, especially for the concentration of cadmium, which decreased by 60–75%. The vermicompost’s influence on changing the chemical form of TMs depended on the TM concentrations, type of TM, and crop species; moreover, blanching effectively reduced the concentrations of TMs in high-mobility chemical forms. Although vermicompost mostly reduced the amount of cadmium consumed via oral intake, cadmium still posed the highest health risk compared to the other three TMs.