Automation is of significant interest to the field of naval ship design and operations, resulting in substantial research, development and implementation efforts being carried out in this field. Many of the expected benefits of automation, such as increased situational awareness and reduced operator workload, have a rather abstract, qualitative nature, and cannot easily be measured. This makes it difficult to predict or evaluate the effectiveness of automation projects. In order to address this challenge, this paper explores to which extent a need for such effectiveness assessment methods is experienced, and what type of information could be provided by such methods. A literature study into methods that are currently available for assessing the effectiveness of automation shows that most methods focus on either quantitative assessments for measuring specific effects, or qualitative assessments for measuring generic effects. Subsequently, a survey with Subject Matter Experts (SME’s) of the Netherlands Ministry of Defence shows that qualitative assessments with user feedback are widely adopted to assess the effectiveness of automation. At the same time, many SME’s recognise the subjective nature of this type of information as a challenge. Based on the literature review and the SME survey, the authors conclude that user feedback, though being subjective, remains an adequate method for assessing the effectiveness of automation, but that additional efforts are needed to apply user feedback effectively in automation programmes. More specifically, there should be an adequate mechanism for dealing with user feedback, as well as opportunity and commitment of all parties involved in the collaboration, in order to make naval automation a success.