The majority of patients today have vascular illness that could cause pulmonary hypertension and pulmonary emboli which can lead to serious conditions. To diagnose changes in veins and arteries, manual, semi-automatic and automatic examination of a individuals chest computed tomography image or CT Pulmonary Angiogram (CTPA) is performed. Manual CT scan analysis is more tedious, takes more time and is not standardized. Thus the separation of vascular trees in CTPA scans is now semi-automatic and automatic which enables doctors to precisely identify aberrant conditions. Recent research has focused on applying machine learning along with deep learning approaches to find and categorize pulmonary vascular disorders like pulmonary embolism and pulmonary hypertension. Here, a brand-new technique is put out for the automatically classifying pulmonary veins in order to more accurately diagnose lung illnesses related to Pulmonary Embolism (PE). The proposed method uses Scaled Siamese-based Convoutional Neural Network (CNN) for classifying Pulmonary Embolism images with an accuracy more than 90 while using two publicly available datasets FUMPE and RSNA. Finally, comparison of the proposed method with other pretrained CNN-based algorithms for detecting PE is done in terms of Area Under Curve (AUC), recall, precision, accuracy, [Formula: see text]1 score, etc. and values are tabulated and our proposed method shows good results. Comparison was also done using deep learning models like PENET, Inception [Formula: see text]3 and [Formula: see text] fold cross validation and our proposed method values are outstanding compared to other methods.