We propose two-dimensional silicon valley phoxonic crystals (VPxCs) to simultaneously exhibit the topological states for electromagnetic and elastic waves. By rotating the trilobal air holes, the coexistence of topological photon and phonon valley Hall phase transitions is achieved due to the breakdown of mirror symmetry. Utilizing this kind of VPxCs, we construct the topological edge states and simultaneously realize topologically valley transmission for both electromagnetic and elastic waves. The simulated results using finite element method show that the proposed VPxCs have robustness against disturbances for both electromagnetic and elastic waves. The acousto-optic (AO) interaction in a topological waveguide-cavity coupled system based on VPxCs is further studied. Results show that obvious shifts of transmission valley of electromagnetic waves during one period of elastic wave. The proposed VPxCs allow for constructing high-performance platforms to realize highly controllable topological photon-phonon interactions with an ultra-small footprint size.