Quantum cutting materials have garnered significant interest due to their potential applications in enhancing photoelectric conversion efficiency in devices such as solar cells. The host matrices that accommodate rare earth ions play a pivotal role in determining the optical transition properties, which are crucial for quantum cutting. In this study, both Er3+ single-doped and Er3+/Yb3+ co-doped LixNa1-xGd(MoO4)2 (x = 0, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 0.99 and 1.0) were synthesized via a high-temperature solid-state reaction to tune the optical transitions properties and enhance quantum cutting efficiency. Optical transition intensity parameters (Judd-Ofelt parameters) of Er3+ were calculated and shown to vary significantly with the Li+/Na+ molar content, demonstrating that these optical properties can be effectively tuned by adjusting the sample composition. The reliability of these calculations was confirmed by comparing the radiative transition rate ratios between two green emissions of Er3 obtained through both theoretical and experimental methods. A two-step energy transfer mechanism was identified as the basis for quantum cutting process, and the relevant energy transfer and nonradiative relaxation rates were determined. Notably, the quantum cutting efficiency reached a peak value of 155.65 % in Li0.5Na0.5Gd(MoO4)2 sample, indicating that successful tuning of the sample's optical transition properties effectively enhanced its quantum cutting efficiency. These findings underscore the viability of tuning the optical properties of rare earth-doped materials to design highly efficient quantum cutting materials.