Superconductivity beyond electron-phonon mechanism is always twisted with magnetism. Based on a new field-effect transistor with solid ion conductor as the gate dielectric (SIC-FET), we successfully achieve an electric-field-controlled phase transition between superconductor and ferromagnetic insulator in (Li,Fe)OHFeSe. A dome-shaped superconducting phase with optimal Tc of 43 K is continuously tuned into a ferromagnetic insulating phase, which exhibits an electric-field-controlled quantum critical behavior. The origin of the ferromagnetism is ascribed to the order of the interstitial Fe ions expelled from the (Li,Fe)OH layers by gating-controlled Li injection. These surprising findings offer a unique platform to study the relationship between superconductivity and ferromagnetism in Fe-based superconductors. This work also demonstrates the superior performance of the SIC-FET in regulating physical properties of layered unconventional superconductors.