Cardiovascular diseases (CVDs) are a group of heart and blood vessel disorders that is one of the most serious dangers to human health, and the number of such patients is still growing. Early and accurate detection plays a key role in successful treatment and intervention. Electrocardiogram (ECG) is the gold standard for identifying a variety of cardiovascular abnormalities. In clinical practices and most of the current research, standard 12-lead ECG is mainly used. However, using a lower number of leads can make ECG more prevalent as it can be conveniently recorded by portable or wearable devices. In this research, we develop a novel deep learning system to accurately identify multiple cardiovascular abnormalities by using only three ECG leads which are I, II, and V1. Specifically, we use three separate One-dimensional Convolutional Neural Networks (1D-CNNs) as backbones to extract features from three input ECG leads separately. The architecture of 1D-CNNs is redesigned for high performance and low computational cost. A novel Lead-wise Attention module is then introduced to aggregate outputs from these three backbones, resulting in a more robust representation which is then passed through a Fully-Connected (FC) layer to perform classification. Moreover, to make the system’s prediction clinically explainable, the Grad-CAM technique is modified to produce a highly meaningful lead-wise explanation. Finally, we employ a pruning technique to reduce system size, forcing it suitable for deployment on hardware-constrained platforms. The proposed lightweight, explainable system is named LightX3ECG. We get classification performance in terms of F1 scores of 0.9718 and 0.8004 on two large-scale ECG datasets, i.e., Chapman and CPSC-2018, respectively, which surpass current state-of-the-art methods while achieving higher computational and storage efficiency. Visual examinations and a sanity check are also performed to strictly demonstrate the strength of our system’s interpretability.