Large and expanding cities suffer from a traffic congestion problem that harms the environment, travelers, and the economy. This paper aims to predict short term traffic congestion on a road section of expressway in Delhi city. For this purpose, we first propose a traffic congestion index based on traffic speed and flow. Clustering techniques and the Greenshield’s model were used for the derivation of the congestion index. Using this congestion index, congested time intervals of each day and each location of a weekday were identified. This study also introduces a feature series long short-term memory neural network (FSLSTMNN), which links a long short-term memory (LSTM) layer to each feature. It is trained using the many heterogeneous traffic features data collected in Delhi city for the next five minutes of traffic flow and speed prediction. FSLSTMNN achieved the good capability to learn feature series data. We also trained several traditional and deep-learning models using the same traffic data. The FSLSTMNN reduces mean absolute error 12.90% and 17.13%, respectively, in speed and traffic flow prediction compared to the second good-performance long short-term memory neural network (LSTMNN). Finally, traffic congestion is predicted classwise (light, medium, and congested) using the developed congestion index and traffic speed and flow predicted by the FSLSTMNN. Predicted results are consistent with the measured field data. Study results confirm that the developed congestion index and FSLSTMNN can be used successfully to predict traffic congestion.