This study introduces an artificial neural network system for ship motion prediction in seaways. To consider the physical characteristics of wave-induced ship motions, neural networks based on a Long Short-Term Memory (LSTM) encoder and decoder, and a convolutional neural network (CNN) are integrated. The LSTM encoder computes the state vector representing the memory effects of motion-induced radiated waves based on past motion records, that is, a sequence-to-one model. In the LSTM decoder, the motion time series is predicted using the encoded initial state vector and foreseen information on the ocean wave field around a vessel, that is, a sequence-to-sequence model. In addition, a CNN is adopted to compress the wave data into a vector sequence. Particularly, the present CNN uses spatiotemporal wave-field data, not a wave signal at single location. To validate the proposed system, a database for training the integrated system was constructed using a physics-based seakeeping program for various sea states. By applying the trained model, deterministic predictions were performed for a new ocean environment, and the accuracy and reliability of the testing results are investigated according to the input data and neural network structures. From the simulation results, it was confirmed that the present encoder–decoder system can conduct ship motion forecasting by effectively considering the motion memory effects and wave excitations as in the ship hydrodynamic model. In addition, excitations and resulting motion responses by short-crested waves can be considered through CNN-based wave-field data processing. Finally, the present machine-learning model also showed the capability of extracting ship operation information (maneuvering quantities) from the given wave-field data.