Temperature has a significant impact on cable-stayed bridges, yielding structural responses comparable to those from vehicular loads, winds, etc. However, advanced numerical techniques for evaluating long-term temperature-induced responses (TIRs) of cable-stayed bridges are complicated and computationally inefficient. Therefore, this study leverages recent advances in deep learning and develops a channel-attention-based bidirectional long short-term memory network (CABLe) to directly get the complex mapping between structural temperatures and TIRs from the monitoring data. The key concept behind is the proposed channel attention mechanism (CAM), where its attention weights are calculated using a cosine similarity between latent sequential features to find the most informative contents of the signal. A comparison study is conducted with the bidirectional long short-term memory (BiLSTM) to show the benefits of the proposed CAM. The proposed method successfully predicts TIRs of a cable-stayed bridge using the imbalanced data. Results indicate that the CABLe outperforms the BiLSTM network and shows a high prediction accuracy with unseen temperature data.