Because of ever-increasing economic globalization, it is necessary to simulate vessel behavior for investigating safety and capacity in ports and inland waterways. A new maritime traffic model was developed; it comprises two parts: the route choice model and the operational model. This paper presents the operational model, which describes vessel sailing behavior by optimal control. In the operational model, the main behavioral assumption is that all actions of the bridge team, such as accelerating and turning, are executed to force the vessel to sail with the desired speed and course. In the proposed theory, deviating from the desired speed and course, accelerating, decelerating, and turning will provide disutility (cost) to the vessel. Through prediction and minimization of this disutility, the longitudinal and angular acceleration can be optimized and predict individual vessel sailing behavior. To verify the route choice model and the operational model, a case study was carried out; it applied the models to predict individual vessel behavior (path, speed, and course) in the entrance channel to Maasvlakte I at the Port of Rotterdam, Netherlands. The simulation results show a good prediction of the vessel path and vessel course. As no other model has been built specifically to predict vessel behavior in the port area, the current methods provide a fundamental basis for investigating vessel behavior in restricted waterways. In addition, this research showed the potential of the model to increase the safety and capacity of ports and inland waterways.