Coordination polymerization of styrene (St) using molybdenum pentachloride supported by phosphite ligand in the presence of metal organic compound was studied for the first time. The types of phosphite and co-catalysts significantly affected the catalytic activity of the molybdenum (V) (Mo(V)) active center and the number-average molecular weight ( Mn) of the resultant polymer. Among the examined catalysts, tri(nonylphenyl)phosphite (TNPP) ligand and AlOPhCH3( i-Bu)2 as co-catalyst provided the polymer with highest yield (up to 87.1%), metallocene as co-catalyst provided the polymer with highest Mn (up to 5.32 × 105). The effect of [P]/[Mo] molar ratio on catalyst activity of the polymerization was discussed and the structures of Mo·TNPP complexes were preliminarily studied by infrared (IR) and ultraviolet spectroscopies. Besides, the polystyrene (PS) samples synthesized through bulk polymerization and solution polymerization were characterized by gel permeation chromatography, IR, carbon 13 nuclear magnetic resonance, and differential scanning calorimetry, respectively, and the results indicated both of the PS had high molecular weight (approximately 105) and atactic structure. All these results demonstrated that Mo(V) catalyst system was very effective for St polymerization.