Anxiety has been increasingly recognized as part of the psychosocial health issues in COVID-19 patients. However, the impact of this topic may be underestimated in low- and middle-income countries. This study aimed to estimate the prevalence of and risk factors of anxiety in COVID-19 patients compared to controls in a local tertiary teaching hospital in Malaysia. In this case-control study, we analyzed data on adult patients aged 18 years and above hospitalized for COVID-19 infection with matched hospitalized controls. The demographic, clinical data and anxiety measures using the Generalized Anxiety Disorder-7 questionnaire were analyzed using univariate and multivariate analysis. 86.6% in the COVID-19 group had anxiety, significantly higher than 13.4% in the control group (p = 0.001). The COVID-19 group was significantly associated with the GAD-7 severity (p = 0.001). The number of COVID-19 patients in the mild, moderate, and severe anxiety groups was 48 (84.2%), 37 (86%), and 18 (94.7%), respectively. Multiple logistic regression showed significant predictors for anxiety, including COVID-19 diagnosis and neurological symptoms. Anxiety was found 36.92 times higher in the patients with COVID-19 compared to those without COVID-19 (OR 36.92;95% CI 17.09, 79.78, p = 0.001). Patients with neurological symptoms were at risk of having anxiety (OR 2.94; 95% CI 1.03, 8.41, p = 0.044). COVID-19 patients experience a significant disruption in psychosocial functioning due to hospitalization. The burden of anxiety is notably high, compounded by a diagnosis of COVID-19 itself and neurological symptomatology. Early psychiatric referrals are warranted for patients at risk of developing anxiety symptoms.