A ternary sol containing silicon, phosphorus and boron modified by γ-methacryloxypropyltrimethoxysilane was synthesized by sol–gel method. The ternary sol was incorporated into the organic matrix and UV-curable organic/inorganic hybrid coating materials were obtained. Hardness, transmittance, haze, cross-cut adhesion and abrasion resistance results showed that the mechanical properties of the hybrid coatings improved effectively with no comprising on optical properties by increasing sol content. Scanning electron microscopy and Energy Dispersive X-ray spectrometer studies indicated that inorganic particles were homogenously dispersed in the organic matrix. The flame retardancy of the UV-curable coatings was investigated by thermogravimetric analysis and microscale combustion calorimeter. The results showed that the incorporation of sol into the organic network led to an improvement in the thermal stability and flame retardancy of the hybrid coating materials. It is a desirable achievement to improve simultaneously both flame retardancy and mechanical properties of the coatings.