The reliable detection of atrial fibrillation (AF) is important for diagnosing the disease, tracking its progression, and developing individualized care strategies. However, models based on limited data are prone to data dependency due to differences in data feature distribution, which generally degrades their performance on unseen external datasets. In this work, we propose a multi-input fusion AF detection network (MIF-AFNet), which cascades residual convolutional neural networks and bidirectional long short-term memory networks to capture the deep features of electrocardiogram (ECG) and RR intervals (RRIs), respectively. Additionally, the ECG signals use a data augmentation method to alleviate the morphological imbalance. MIF-AFNet learns a robust feature representation for accurate AF detection by fusing the available information from RRIs and ECG. The proposed method was developed and evaluated using 5 long-term ECG datasets (CPSC2021, AFDB, LTAF, MITDB, and NSRDB) from PhysioNet. The subject-wise five-fold cross-validation was performed on CPSC2021, and the proposed method achieved an AF detection accuracy of 98.63%. The generalization performance is further evaluated on four external independent datasets (AFDB, LTAF, MITDB, and NSRDB), achieving accuracies of 98.63%, 97.04%, 98.07%, and 100%, respectively. The results show that the proposed method can accurately detect AF from long-term ECG recordings. In addition, the low complexity of the model makes it less demanding on computing resources. Therefore, it has the potential to improve the automatic diagnosis and management of AF in wearable device-based long-term home monitoring.