Adverse weather conditions like fog, rainfall, and snowfall affect the driver’s visibility, mobility of vehicle, and road capacity. Accurate prediction of the macroscopic traffic stream variables such as speed and flow is essential for traffic operation and management in an Intelligent Transportation System (ITS). The accurate prediction of these variables is challenging because of the traffic stream’s non-linear and complex characteristics. Deep learning models are proven to be more accurate for predicting traffic stream variables than shallow learning models because it extracts hidden abstract representation using layerwise architecture. The impact of weather conditions on traffic is dependent on various hidden features. The rainfall effect on traffic is not directly proportional to the distance between the weather station and the road because of terrain feature constraints. The prolonged rainfall weakens the drainage system, affects soil absorption capability, which causes waterlogging. Therefore, to capture the spatial and prolonged impact of weather conditions, we proposed a soft spatial and temporal threshold mechanism. To fill out the missing weather data spatial interpolation techniques are used. The traffic condition on a target road depends on the surrounding area’s traffic and weather conditions and relies on its own traffic characteristics. We designed the hybrid deep learning models, CNN-LSTM and LSTM-LSTM. The former model in the hybrid model extracts the spatiotemporal features and the latter model uses these features as memory. The latter model predicts the traffic stream variables depending upon the passed features and temporal input. We perform multiple experiments to validate the deep learning model’s performance. The experiments show that a deep learning model trained with traffic and rainfall data gives better prediction accuracy than the model trained without rainfall data. The performance of the LSTM-LSTM model is better than other models in extracting long-term dependency between the traffic and weather data.