Semantic segmentation of small-scale objects in aerial images is challenging due to low-level characteristics in neural networks and varying information in feature maps. Deeplab models struggle with poor edge refinement, resulting in rough borders, and fail to fully exploit relationships between pixel categories at different distances. To addressing the issue in deeplab series, an adaptive multi-level attention based deeplabv3+ (AMLA-Deeplabv3+) with improved golden jackal optimization algorithm is implemented in this paper. Multi-level attention unit has been included in the Atrous spatial pyramid pooling module in the encoder section of deeplabv3+ to bridge the semantic feature gap among encoders output. To put more weights on relevant features squeeze and excitation units has been included in the decoder section of deeplabv3+. The improved golden jackal optimization (IGJO) algorithm is deployed to fine tune the hyper parameters such as number of hidden neurons, epochs, learning rate and batch size of deeplabv3+. The proposed AMLA-Deeplabv3+ is tested on two publicly available datasets named semantic segmentation of aerial imagery and aerial image segmentation. The proposed model achieves better segmentation outcomes in terms of accuracy of 99.65%, Precision of 97.21%, F1-score of 99.48% and mean intersection over union of 98.44% with a computation time of 3.0231 minutes. The experimental outcomes shows that the proposed model’s efficiency is superior to other conventional techniques.