The COVID-19 pandemic becomes a cause of concern for hospital transmission. Caregivers may play an important role as vectors for nosocomial infections; however, infection control for caregivers often is neglected. A nosocomial COVID-19 outbreak occurred in a 768-bed hospital from March 20, 2020, to April 14, 2020. We conducted a retrospective chart review and epidemiologic investigation on all cases. A total of 54 cases of laboratory-confirmed COVID-19 occurred in the community-based hospital. They included 26 (48.1%) patients, 21 (38.9%) caregivers, and 7 (13.0%) healthcare workers. These 21 caregivers cared for 18 patients, and of these, 9 were positive for COVID-19, 6 were negative, and 3 died before testing. Of the 6 negative patients, 3 had no exposure because the caregiver began to show symptoms at least 5 days after their discharge. Of the 9 positive patients, 4 cases of transmission took place from patient to caregiver (one patient transmitted COVID-19 to two caregivers), and 6 cases of transmission occurred from caregiver to patient. Of the 54 hospital-acquired cases, 38 occurred in the 8th-floor ward and 8 occurred in the 4th-floor ward. The index case of each ward was a caregiver. Counting the number of cases where transmission occurred only between patients and their own caregivers, 9 patients were suspected of having exposure to COVID-19 from their own caregivers. Six patients (66.7%) were infected by COVID-19-confirmed caregivers, and 3 patients were uninfected. Fewer patients among the infected were able to perform independent activities compared to uninfected patients. Not only patients and healthcare workers but also caregivers groups may be vulnerable to COVID-19 and be transmission sources of nosocomial outbreaks. Therefore, infection control programs for caregivers in addition to patients and healthcare workers can be equally important.