We theoretically propose an optical polarization-state engineering based on a cavity optomagnonic platform in the full quantum regime. Here, the optomagnonic interaction can be realized in an yttrium iron garnet (YIG) sphere, which supports both two traveling-photon modes with orthogonal linear polarizations [i.e., horizontally ($H$-) and vertically ($V$-) polarized photon modes] and a Kittel magnon mode. In our scheme, the magnons mediate the photons during polarization conversions, and break the time-reversal symmetry in mutual interconversions. Through a triple resonance between the magnon mode, $H$-polarized, and $V$-polarized photon modes within the YIG optomagnonic cavity, we demonstrate an all-optical scheme to manipulate the optical polarization behaviors by adjusting an external driving laser, finding that the polarization states of the output light can be well mapped to the whole Poincar\'e sphere and a large polarization rotation of the output light with photon antibunching and superbunching can be achieved easily for a range of realistic parameters. We find the strong discrepancy in the polarization response between the exact numerical calculation using the full quantum master equation and the semiclassical approximation. In addition, we reveal the magnon-induced broken time-reversal symmetry, which connects the quantum cavity optomagnonics to the classical magneto-optical effect. Our obtained results have potential application in quantum polarization-state engineering, and also offer a further understanding for cavity optomagnonics at the quantum level.