All over the world, many ports have implemented surveillance camera systems to monitor the vessels and activities around them. These types of systems are not very effective in accurately detecting activities around the port due to background noise and congestion interference at the sea surface. This is why it is difficult to accurately detect vessels, especially smaller vessels, when it turns dark. It is known that some vessels do not comply with maritime rules, particularly in port and safety zones; these must be detected to avoid incidents. For these reasons, in this study, we propose and develop an improved multi-structural morphology (IMSM) approach to eliminate all of this noise and interference so that vessels can be accurately detected in real time. With this new approach, the target vessel is separated from the sea surface background through the weighted morphological filtering of several datasets of structural components. Then, neighborhood-based adaptive fast median filtering is used to filter out impulse noise. Finally, a characteristic morphological model of the target vessel is established using the connected domain; this allows the sea surface congestion to be eliminated and the movement of vessels to be detected in real time. Multiple tests are carried out on a small and discrete area of moving vessels. The results from several collected datasets show that the proposed approach can effectively eliminate background noise and congestion interference in video monitoring. The detection accuracy rate and the processing time are improved by approximately 3.91% and 1.14 s, respectively.