Low temperature is a great challenge for the biological treatment of wastewater. In this study, the rapid start-up of aerobic granular biofilm (AGF) reactor was realized by adding micro-sized polyurethane (PU) sponges as matrices at 10 °C. The results showed that the granulation process of AGF was different from that of traditional aerobic granular sludge and biofilms, which was formed by using the sludge intercepted in PU matrix instead of sponge skeletons as granulation carriers. During the 5-month operation period, stable pollutants removal performance was achieved within 70 days, besides, the corresponding ammonium, total nitrogen, and total phosphorus removal efficiencies were 98%, 70%, and 95%, respectively. The addition of PU matrices inhibited the growth of filamentous bacteria and provided support for high structural stability of AGF. With the operation of the reactor, the relative abundance of traditional denitrifying bacteria (genera Thauera and Acidovorax, etc.) decreased gradually, and the putative denitrifying phosphorus accumulating genus, Dechloromonas, occupied a dominant position in the system. This experiment showed that AGF system could be successfully started-up and operated with efficient pollutants removal performance under low temperature when using micro-sized PU sponges as matrices.