The methods of intelligent port container number detection tend to be diversified. However, traditional container number positioning methods and current deep learning algorithms are mostly multi-stages, which need to be optimized during difficultly training, resulting in bad model effect and time-consuming. In view of the above problems, the main contributions of this paper are drawing on the EAST text detection method to obtain an improved EAST algorithm that directly predicts the box number area. This new algorithm not only eliminates multiple intermediate stages, but also improves the regression method and loss function of the box number area, optimizes the boundary, balances positive and negative samples to adapt to the detection area of the box number based on the original algorithm. At the same time, it adopts the lightweight design idea, and uses ShuffleNet’s channel shuffling and depth separable convolution for reference to optimize the original model, which reduces the complexity of time and space, compresses the model and reduces the detection time. The final experimental results show that the accuracy of the algorithm reaches 97.5% while keeping the FPS index no less than 14.