Octogen (HMX) is the most powerful military explosive in current use. However, the unclear thermodynamic molecular mechanisms severely limit the development of its production process. In this paper, the solubility of β-HMX in three binary mixed solvent systems (acetone + ethanol, acetone + dichloromethane, acetone + toluene) was measured via a dynamic method at temperature ranging from 283.15 to 313.15 K under atmospheric pressure. The experimental results reveal that the solubility of β-HMX in these binary mixed solvent systems is positively correlated with acetone molar fraction and temperature, and the solvent composition has a greater impact on its solubility compared with the temperature. Moreover, the solubility of β-HMX is further correlated with the Van’t Hoff equation, modified Apelblat equation, CNIBS/R-K model, Jouyban-Acree-Van’t Hoff model, Jouyban-Acree-Apelblat model and NRTL model, and the modified Apelblat equation can give better fitting result (ARD = 8.101 × 10−3, RMSD = 1.065 × 10−5). Finally, the microscopic mechanism of the thermodynamic properties of β-HMX in binary mixed solvents is revealed by investigating the types, intensity and interaction sites of interactions based on density functional theory (DFT) and molecular dynamic (MD) simulation. The results demonstrate that both acetone molecules and ethanol molecules can form certain intermolecular interactions with β-HMX molecules, which are the main driving force of solubilization. In contrast, solvent–solvent interactions have an antagonistic effect on the solubility of β-HMX in mixed solvents. This work can further be informative for researchers in the field of energetic material for the solubility evaluations and provide a reference for the industrial production and performance enhancement of HMX in the future.