The automatic detection and analysis of sunspots play a crucial role in understanding solar dynamics and predicting space weather events. This paper proposes a novel method for sunspot group detection and classification called the dual stream Convolutional Neural Network with Attention Mechanism (DSCNN-AM). The network consists of two parallel streams each processing different input data allowing for joint processing of spatial and temporal information while classifying sunspots. It takes in the white light images as well as the corresponding magnetic images that reveal both the optical and magnetic features of sunspots. The extracted features are then fused and processed by fully connected layers to perform detection and classification. The attention mechanism is further integrated to address the “edge dimming” problem which improves the model’s ability to handle sunspots near the edge of the solar disk. The network is trained and tested on the SOLAR-STORM1 data set. The results demonstrate that the DSCNN-AM achieves superior performance compared to existing methods, with a total accuracy exceeding 90%.