The automation transformation of container lifting operations is one of the main technical issues in Rail-Truck intermodal transportation. To solve this problem, this paper analyzes the advantages and disadvantages of several existing container positioning methods, and proposed a container corner holes location detection method based on lightweight convolutional neural network and adaptive morphological image processing algorithm. This method locates the container through a visual sensor that shoots from top to bottom. In order to improve the positioning accuracy and calculation speed while maintaining a high recognition rate, this method first uses a lightweight SSD detector to quickly detect the rough coordinates of the container corner hole in the image. On this basis, the smallest rectangle detection method based on adaptive HSV filtering is used to detect the precise coordinates of the corner hole in the image. Experiments show that the recognition rate of this algorithm in the initial positioning process reaches 94.2%, the recognition rate of secondary positioning reaches 87.4%, and the final positioning error is lower than 3.765 pixels and 2.81 pixels in the heading and lateral directions, respectively. The overall calculation time of the algorithm can realize the positioning calculation of 30 frames per second, which shows that this method takes into account the characteristics of high measurement accuracy and high calculation speed on the basis of high recognition rate.