Modulation signal classification in communication systems can be considered a pattern recognition problem. Earlier works have focused on several feature extraction approaches such as fractal feature, signal constellation reconstruction, etc. The recent advent of deep learning (DL) models makes it possible to proficiently classify the modulation signals. In this view, this study designs a chaotic oppositional satin bowerbird optimization (COSBO) with bidirectional long term memory (BiLSTM) model for modulation signal classification in communication systems. The proposed COSBO-BiLSTM technique aims to classify the different kinds of digitally modulated signals. In addition, the fractal feature extraction process takes place by the use of Sevcik Fractal Dimension (SFD) approach. Moreover, the modulation signal classification process takes place using BiLSTM with fully convolutional network (BiLSTM-FCN). Furthermore, the optimal hyperparameter adjustment of the BiLSTM-FCN technique takes place by the use of COSBO algorithm. In order to ensure the enhanced classification performance of the COSBO-BiLSTM model, a wide range of simulations were carried out. The experimental results highlighted that the COSBO-BiLSTM technique has accomplished improved performance over the existing techniques.