Quantum dots (QDs) are widely utilized semiconductor nanocrystal materials with both nanotoxicity and composition-related toxicity. To determine the toxicological impacts and underlying mechanisms of QDs with different compositions on microalgae, carbon QDs (CQDs) and CdSe QDs were used in the present study. Results showed that QDs composed of CdSe were more toxic than QDs composed of carbon, which inhibited cell growth, with reductions in chl b content, chlorophyll fluorescence parameters, and increases in lipids and starch (two major storage substances). In addition, CdSe QDs elevated reactive oxygen species (ROS), resulting in oxidative damage, while CQDs had little effect on antioxidants. Comparative transcriptome analysis showed that gene expression was accelerated by CdSe QDs, and there was a compensatory upregulation of porphyrin metabolism, potentially to support chlorophyll synthesis. In addition, an MYB transcription factor was predicted by weighted gene co-expression network analysis (WGCNA) to serve as regulator in nanoparticle toxicity, while glutathione peroxidase (GPX) and dual-specificity tyrosine phosphorylation regulated kinases 2/3/4 (DYRK2/3/4) may be key mediators of the composition-related toxicity of CdSe QDs. This study highlights the critical role of QDs’ composition in determining their impacts on aquatic microalgae, providing a theoretical reference for selecting appropriate QDs materials for various industrial applications.