An amorphous mixing layer (3.5–4.0 nm in thickness) containing silicon (Si), oxygen (O), molybdenum (Mo) atoms, named <i>α</i>-SiO<sub><i>x</i></sub>(Mo), is usually formed by evaporating molybdenum trioxide (MoO<sub>3</sub>) powder on an n-type Si substrate. In order to investigate the process of adsorption, diffusion and nucleation of MoO<sub>3</sub> in the evaporation process and ascertain the formation mechanism of <i>α</i>-SiO<sub><i>x</i></sub>(Mo) on a atomic scale, the first principle calculation is used and all the results are obtained by using the Vienna <i>ab initio</i> simulation package. The possible adsorption model of MoO<sub>3</sub> on the Si (100) and the defect formation energy for substitutional defects and vacancy defects in <i>α</i>-SiO<sub>2</sub> and <i>α</i>-MoO<sub>3</sub> are calculated by the density functional theory. The results show that an amorphous layer is formed between MoO<sub>3</sub> film and Si (100) substrate according to <i>ab initio</i> molecular dynamics at 1500 K, which are in good agreement with experimental observations. The O and Mo atoms diffuse into Si substrate and form the bonds of Si—O or Si—O—Mo, and finally, form an <i>α</i>-SiO<sub><i>x</i></sub>(Mo) layer. The adsorption site of MoO<sub>3</sub> on the reconstructed Si (100) surface, where the two oxygen atoms of MoO<sub>3</sub> bond with two silicon atoms of Si (100) surface, is the most stable and the adsorption energy is -5.36 eV, accompanied by the electrons transport from Si to O. After the adsorption of MoO<sub>3</sub> on the Si substrate, the structure of MoO<sub>3</sub> is changed. Two Mo—O bond lengths of MoO<sub>3</sub> are 1.95 Å and 1.94 Å, respectively, elongated by 0.22 Å and 0.21 Å compared with the those before the adsorption of MoO<sub>3</sub> on Si substrate, while the last bond length of MoO<sub>3</sub> is little changed. The defect formation energy value of neutral oxygen vacancy in <i>α</i>-SiO<sub>2</sub> is 5.11 eV and the defect formation energy values of neutral oxygen vacancy in <i>α</i>-MoO<sub>3</sub> are 0.96 eV, 1.96 eV and 3.19 eV, respectively. So it is easier to form oxygen vacancy in MoO<sub>3</sub> than in SiO<sub>2</sub>, which implies that the oxygen atoms will migrate from MoO<sub>3</sub> to SiO<sub>2</sub> and forms a 3.5–4.0-nm-thick <i>α</i>-SiO<sub><i>x</i></sub>(Mo) layer. As for the substitutional defects in MoO<sub>3</sub> and SiO<sub>2</sub>, Mo substitutional defects are most likely to form in SiO<sub>2</sub> in a large range of Mo chemical potential. So based on our obtained results, the forming process of the amorphous mixing layer may be as follows: the O atoms from MoO<sub>3</sub> bond with Si atoms first and form the SiO<sub><i>x</i></sub>. Then, part of Mo atoms are likely to replace Si atoms in SiO<sub><i>x</i></sub>. Finally, the ultra-thin buffer layer containing Si, O, Mo atoms is formed at the interface of MoO<sub>3</sub>/Si. This work simulates the reaction of MoO<sub>3</sub>/Si interface and makes clear the interfacial geometry. It is good for us to further understand the process of adsorption and diffusion of atoms during evaporating, and it also provides a theoretical explanation for the experimental phenomenon and conduces to obtaining better interface passivation and high conversion efficiency of solar cell.