Ventricular arrhythmias are the primary arrhythmias that cause sudden cardiac death. We address the problem of classification between ventricular tachycardia (VT), ventricular fibrillation (VF) and non-ventricular rhythms (NVR). To address the challenging problem of the discrimination between VT and VF, we develop similarity maps - a novel set of features designed to capture regularity within an ECG trace. These similarity maps are combined with features extracted through learnable Parzen band-pass filters and derivative features to discriminate between VT, VF, and NVR. To combine the benefits of these different features, we propose a hierarchical multi-stream ResNet34 architecture. Our empirical results demonstrate that the similarity maps significantly improve the accuracy of distinguishing between VT and VF. Overall, the proposed approach achieves an average class sensitivity of 89.68%, and individual class sensitivities of 81.46% for VT, 89.29% for VF, and 98.28% for NVR. The proposed method achieves a high accuracy of ventricular arrhythmia detection and classification. Correct detection and classification of ventricular fibrillation and ventricular tachycardia are essential for effective intervention and for the development of new therapies and translational medicine.