Let the robot tell: Describe car image with natural language via LSTM

Abstract

Image-based car detection and classification has remained as a research hub in self-driving for decades. However, natural language description of car images is still a virgin territory even though it is a simple task for human to describe it by sentences within a glimpse at the image. In this paper, we present an end-to-end trainable and spatial-temporal deep recurrent neural network: LSTM (Long-Short Term Memory) to automatically convert car images to human understandable natural language descriptions. Our model builds on state of the art progress in computer vision and machine translation: we extract car region proposals with Region Convolutional Neural Networks(R-CNN) and embed them into fixed-sized vectors. Each word in a sentence is also embedded into real-valued vectors of the same size of images through a local global context aware neural network. The LSTM, feeding by image-sentence pairs sequentially in the training stage, is trained to maximize the joint probability of target word in each time step. In the test stage, the pre-trained LSTM receives a car image and predicts natural language description word by word. Finally, we evaluate our model regarding car’s static/dynamic attribute description on both 30,000 CompCar dataset [21] and 1000 video dataset collected on street scenario by our self-driving car, with quantitative BLEU score and subjective human-rating system evaluation metrics. We test our model’s generalization ability, its transfer ability to address car property classification issue and various image feature extractors’ impact on our model. Experiment results show the superiority and robustness of our model (refer to www.carlib.net/carimg2text.html for more experiment results).