The coastal zone is connected to the hinterland of the basin and the wide sea, which is not only affected by complex natural factors, but also by human activities. The study of sediment classification in this area will help to further explain topographic evolution and dynamic mechanism. Aiming at the complex mixed sediment environment in the ancient valley area of shallow seas, an advanced deep convolutional neural network classification model based on transfer learning—selective kernel hybrid dilated ResNet-50 (SKHD-ResNet-50) is thus proposed and applied to the coastal area in the Firth of Forth, Scotland, UK. The model overcomes problems of low accuracy of sediment classification with small samples, while effectively improving the efficiency of mixed sediment classification. Five transferred convolutional neural network models including CNN, AlexNet, VGG-19, GoogleNet, and SKHD-ResNet-50, were used to conduct seabed sediment classification experiments on multibeam backscatter mosaics in the study area. An ablation study about the effect of selective kernel convolution (SKC) and hybrid dilated convolution (HDC) was also conducted to further prove the usability of the proposed method. Finally, the optimized convolutional neural network was used to build a multi-layer deep learning model for sediment classification, combining the transfer learning method with the neural network to obtain the optimal network weights through deep global optimization, thus accurately mining statistical characteristics and distribution rules of backscatter intensity data. The experimental results show that the advanced model obtains better classification accuracy than that of state-of-art methods, with overall accuracy and Kappa coefficient of 96.87% and 0.9482, respectively. Findings highlight the proposed sediment classification model, which can effectively distinguish mixed sediments, and greatly improve the accuracy and efficiency of sediment classification.