Atrial fibrillation (AF) and atrial flutter (AFL) are the most common arrhythmias. Due to the similar clinical symptoms, both are one of the main causes of misdiagnosis for physicians. The visual inspection of electrocardiogram (ECG) signals is the most traditional detection strategy, however, it is often laborious and time-consuming. Thus, we specially propose a modified bidirectional long short-term memory (MB-LSTM) network for AF and AFL signals recognition. In this network, inspired by illustrious Squeeze-and-Excitation network, a feature recalibration strategy is performed on existing B-LSTM network so that this can enable the model to adaptively reallocate feature representation and thus alleviate the problem of information redundancy in B-LSTM to a certain extent. Further, we embed the proposed network into a convolutional network frame with attention mechanism and use existing LSTM, B-LSTM networks as control groups to evaluate its effectiveness with a subject-independent validation strategy on the two publicly available databases. The result shows that the model yields superior classification performance with an accuracy of 99.1% and 98.4% than several state-of-the-art methods while confirming the effectiveness of MB-LSTM. Particularly, the qualitative analysis is provided to elaborate on the mechanism of performance improvement, showing promising model practicability as an intelligent and efficient tool to assist physicians.