Vulnerable plaque detection is important to acute coronary syndrome (ACS) diagnosis. In recent years, intravascular optical coherence tomography (IVOCT) imaging has been used for vulnerable plaque detection. Current automated detection methods adopt the traditional image classification and object detection algorithms, such as the logistic regression model, SVM, and Haar-Adaboost, to detect vulnerable plaques. The detection quality of these methods is relatively low. The aim of this study is to improve the detection quality of vulnerable plaque. We propose an automatic detection system of vulnerable plaque for IVOCT images based on deep convolutional neural network (DCNN). The system is mainly composed of four modules: pre-processing, deep convolutional neural networks (DCNNs), post-processing, and ensemble. The IVOCT images input to DCNNs are firstly pre-processed by using the methods of de-noising and data augmentation. Then multiple DCNNs are used to detect the vulnerable plaques in the IVOCT images; the vulnerable plaque regions and their corresponding labels and scores are output. Next, the output results of each network are processed by the post-processing module. We propose three algorithms, union of intersecting regions, duplicated region processing, and small gaps removal for post-processing. Finally, the output detection results of multiple networks are combined using a proposed combining method in ensemble module. We evaluated the proposed method in a dataset of 300 IVOCT images. Experimental results show that our system can achieve a precision rate of 88.84%, a recall rate of 95.02%, and an overlap rate of 85.09%; the detection quality score is 88.46%. The proposed algorithms can detect vulnerable plaques with superior performance; our system can be used as an auxiliary diagnostic tool for vulnerable plaque detection in IVOCT images.