Studying the mental effort in problem-solving is important to the understanding of how the brain allocates cognitive resources to process information. The electroencephalogram is a promising physiological approach to assessing the online mental effort. In this study, we investigate the EEG indicators of mental effort while solving scientific problems. By manipulating the complexity of the scientific problem, the level of mental effort also changes. With the increase of mental effort, theta synchronization in the frontal region and lower alpha desynchronization in the parietal and occipital regions significantly increase. Also, upper alpha desynchronization demonstrates a widespread enhancement across the whole brain. According to the functional topography of brain activity in the theta and alpha frequency, our results suggest that the mental effort while solving scientific problems is related to working memory, visuospatial processing, semantic processing and magnitude manipulation. This study suggests the reliability of EEG to evaluate the mental effort in an educational context and provides valuable insights into improving the problem-solving abilities of students in educational practice.