Biogenic volatile organic compounds (BVOCs) are the largest source of secondary organic aerosols (SOA) globally. However, the complex interactions between marine and terrestrial BVOCs remain unclear, inhibiting our in-depth understanding of the SOA formation in the coastal areas and its environmental impacts. Here, we performed smog chamber experiments with mixed α-pinene (a typical monoterpene) and dimethyl sulfide (DMS, a typical marine emission BVOC) to investigate their possible interactions and subsequent SOA formation. It is found that DMS has a non-linear effect on SOA generation: The mass concentration and yield of SOA show increasing and then decreasing trends with the increase of the initial concentration of DMS. The increasing trend can be attributed to OH regeneration from isomerization of the CH3SCH2OO radical together with acid-catalyzed heterogeneous reactions by the oxidation of DMS, while the decreasing trend is explained by the less contribution of isomerization reaction and the high OH reactivity that inhibits the formation of low volatility products. The results from infrared spectra and mass spectra together reveal the contribution of sulfur-containing molecules in the mixed system. Moreover, the mass spectra results indicate that acidic products generated by DMS photooxidation enhance the O:C ratio, while organosulfates are produced to contribute to the formation of mixed SOA. In addition, the trends in relative abundance of highly oxygenated organic molecules (HOMs) with C8 - C10 multiple functional groups in different mixed systems agree well with the turning point of the SOA yield. The findings of this study have significant implications for understanding binary or more complex systems in the atmosphere in the coastal areas.