Diffusion magnetic resonance imaging (dMRI) currently stands as the foremost noninvasive method for quantifying brain tissue microstructure and reconstructing white matter fiber pathways. However, the inherent free diffusion motion of water molecules in dMRI results in signal decay, diminishing the signal-to-noise ratio (SNR) and adversely affecting the accuracy and precision of microstructural data. In response to this challenge, we propose a novel method known as the Multiscale Fast Attention-Multibranch Irregular Convolutional Neural Network for dMRI image denoising. In this work, we introduce Multiscale Fast Channel Attention, a novel approach for efficient multiscale feature extraction with attention weight computation across feature channels. This enhances the model's capability to capture complex features and improves overall performance. Furthermore, we propose a multi-branch irregular convolutional architecture that effectively disrupts spatial noise correlation and captures noise features, thereby further enhancing the denoising performance of the model. Lastly, we design a novel loss function, which ensures excellent performance in both edge and flat regions. Experimental results demonstrate that the proposed method outperforms other state-of-the-art deep learning denoising methods in both quantitative and qualitative aspects for dMRI image denoising with fewer parameters and faster operational speed.