Osteosarcomas are a type of bone tumour that can develop anywhere in the bone but most typically do so around the metaphyseal growth plates at the ends of long bones. Death rates can be lowered by early detection. Manual osteosarcoma identification can be difficult and requires specialised knowledge. With the aid of contemporary technology, medical photographs may now be automatically analysed and categorised, enabling quicker and more effective data processing. This paper proposes a novel hyperparameter-tuned deep learning (DL) approach for predicting osteosarcoma on histology images with effective feature selection mechanism which aims to improve the prediction accuracy of the classification system for bone tumor detection. The proposed system mainly consists of ‘6’ phases: data collection, preprocessing, segmentation, feature extraction, feature selection, and classification. Firstly, the dataset of histology images is gathered from openly available sources. Then Median Filtering (MEF) is utilized as the preprocessing step that enhances the quality of the input images for accurate prediction by eliminating unwanted information from them. Afterwards, the pre-processed image was segmented using Harmonic Mean-based Otsu Thresholding (HMOTH) approach to obtain the tumor-affected regions from the pre-processed data. Then the features from the segmented tumor portions are extracted using the Self-Attention Mechanism-based MobileNet (SAMMNet) model. A Van der Corput sequence and Adaptive Inertia Weight included Reptile Search Optimization Algorithm (VARSOA) is used to select the more relevant features from the extracted features. Finally, a Hyperparameter-Tuned Deep Elman Neural Network (HTDENN) is utilized to diagnose and classify osteosarcoma, in which the hyperparameters of the neural network are obtained optimally using the VARSOA. The proposed HTDENN attains the higher accuracy of 0.9531 for the maximum of 200 epochs, whereas the existing DENN, MLP, RF, and SVM attains the accuracies of 0.9492, 0.9427, 0.9413, and 0.9387. Likewise, the proposed model attains the better results for precision (0.9511), f-measure (0.9423), sensitivity (0.9345) and specificity (0.9711) than the existing approaches for the maximum of 200 epochs. Simulation outcomes proved that the proposed model outperforms existing research frameworks for osteosarcoma prediction and classification.