In this paper, we demonstrate that Deep Learning (DL) and spatiotemporal reasoning can effectively identify driving behavior based on the videos captured by roadside cameras. The use of roadside infrastructure for such determination is twofold: (1) a global view of the vehicles and their interactions, and (2) no involvement or awareness of the vehicles or their drivers, so the determination is inexpensive, easy to deploy, and entirely non-intrusive. Furthermore, our method uses deep learning only for object detection and tracking and builds a flexible and explainable reasoning model to identify the driving behavior. The essential advantage of this approach is that we use deep learning only for tasks that can be accomplished efficiently and with high accuracy (i.e., object detection and tracking), which can be done in real-time. Although there are deep learning models for detecting complex activities (e.g., aggressive driving), they are much harder to train, require higher accuracy, and inferencing time may not satisfy real-time constraints. By using a setup with program-controlled robocars, we demonstrate that we can achieve accuracies of 98–99% for driving behavior characterization, and the mechanism can provide detection of 650 ms on a very dated desktop. The characterization can provide feedback to the driver (or the automated car) for improved traffic safety and roadway throughput.