The application of floating treatment wetlands (FTWs) in point and non-point source pollution control has received much attention recently. Although the potential of this emerging technology is supported by various studies, quantifying FTW performance in urban retention ponds remains elusive due to significant research gaps. Actual urban retention pond water was utilized in this mesocosm study to evaluate phosphorus and nitrogen removal efficiency of FTWs. Multiple treatments were used to investigate the contribution of each component in the FTW system with a seven-day retention time. The four treatments included a control, floating mat, pickerelweed (Pontederia cordata L.), and softstem bulrush (Schoenoplectus tabernaemontani). The water samples collected on Day 0 (initial) and 7 were analyzed for total phosphorus (TP), total particulate phosphorus, orthophosphate, total nitrogen (TN), organic nitrogen, ammonia nitrogen, nitrate-nitrite nitrogen, and chlorophyll-a. Statistical tests were used to evaluate the differences between the four treatments. The effects of temperature on TP and TN removal rates of the FTWs were described by the modified Arrhenius equation. Our results indicated that all three FTW designs, planted and unplanted floating mats, could significantly improve phosphorus and nitrogen removal efficiency (%, E-TP and E-TN) compared to the control treatment during the growing season, i.e., May through August. The E-TP and E-TN was enhanced by 8.2% and 18.2% in the FTW treatments planted with the pickerelweed and softstem bulrush, respectively. Organic matter decomposition was likely to be the primary contributor of nutrient removal by FTWs in urban retention ponds. Such a mechanism is fostered by microbes within the attached biofilms on the floating mats and plant root surfaces. Among the results of the four treatments, the FTWs planted with pickerelweed had the highest E-TP, and behaved similarly with the other two FTW treatments for nitrogen removal during the growth period. The temperature effects described by the modified Arrhenius equation revealed that pickerelweed is sensitive to temperature and provides considerable phosphorus removal when water temperature is greater than 25 °C. However, the nutrient removal effectiveness of this plant species may be negligible for water temperatures below 15 °C. The study also assessed potential effects of shading from the FTW mats on water temperature, DO, pH, and attached-to-substrate periphyton/vegetation.