We propose integrating conversational user interfaces (CUIs) into Maritime Autonomous Surface Ships (MASS) to facilitate direct communication with other ships without human intervention. The aim is to enable other ships and shore stations to inquire about and discuss the autonomous vessel's status, decisions, and future intentions in natural language, offloading VHF radio conversations for MASS operators. Recent advancements in Natural Language Processing (NLP) technology show promise. This paper presents an experimental study conducted in a ship simulator using a functional prototype CUI system. While the CUI system was capable of providing coherent answers to questions, the results indicate that interacting with the system is less trustworthy than interacting with a human ship officer. Key challenges contributing to this included: (1) a missing connection between the CUI and an autonomy system, leading to inaccurate and incorrect answers, (2) speech that lacked intonation and human-like qualities, and (3) a perceived inability of the system to collaborate. The paper discusses several aspects of utilising CUI systems in a maritime shipping environment and concludes by proposing an approach to develop and implement them safely.