Summary Physical abuse in care settings is a social problem with detrimental effects for residents, but effective prevention is dependent on staff reporting. In Sweden, designated officials, commonly social workers, are required to investigate reports of mistreatment. The aim of this study was to analyze what designated officials considered serious in reported incidents of physical abuse in two different care settings: care for older people and care for people with intellectual disabilities. Similar cases of physical abuse were chosen to demonstrate differences between judgments and provide a picture of how designated officials judge the relative seriousness of abusive situations. Interpretative content analysis and analytic induction were used to analyze reports of staff-to-resident abuse and resident-to-resident abuse in the two care areas. Findings There were no differences between incidents described as serious and those described as non-serious. Resident-to-resident abuse was the most commonly reported incident, particularly in dementia care and care for people with intellectual disabilities, and was rarely considered serious. Older people in dementia care were the highest risk group. In staff-to-resident abuse, staff were singled out as scapegoats. Organizational issues, rather than the details of abusive incidents, guided investigators’ judgments of seriousness. Tendencies to normalize violence and apply a narrow understanding of violence were found in the investigator’s judgments, particularly in resident-to-resident abuse. Applications Safeguarding adults from physical abuse in institutional settings is dependent on staff reporting and thorough investigations. This study shows that more training in assessing violence for social workers investigating reports is needed.