Poisonous plants are the third largest category of poisons known globally, which pose a risk of poisoning and death to humans. Currently, the identification of medicinal and poisonous plants is done by humans using experimental methods, which are not accurate and are associated with many errors, and also the use of laboratory methods requires experts and this method is very costly and time-consuming. Therefore, distinguishing between medicinal and poisonous plants is very important using emerging, non-destructive, fast and accurate methods such as computer vision and artificial intelligence. In this study, we propose a robust and generalized model using spatial attention (SA) and channel attention (CA) modules for the classification of different plants. A dataset containing 900 confirmed images of three plant classes (oregano, poisonous and weed) was used. The attention mechanisms enhance efficiency of deep learning (DL) networks by allowing them to precisely focus on all relevant input elements. In order to enhance the performance of the proposed model, the CA was implemented based on four pooling operations including global average pooling-based CA (GAP-CA), mixed pooling-based CA (Mixed-CA), gated pooling-based CA (Gated-CA), and tree pooling-based CA (Tree-CA) operations. The results showed that the DL model based on Tree-CA had promising performance and outperformed other state-of-the-art models, achieving the values of 99.63%, 99.38%, 99.52%, 99.74%, and 99.42%, for accuracy, precision, recall, specificity, and F1-score, respectively. The findings support our proposed attention model's success in identifying medicinal plants from similar poisonous plants. Recent advancements in computer-based technologies and artificial intelligence enable automatic detection of medicinal and poisonous plants, revolutionizing traditional identification methods.