This study analyzed online public discourse on Twitter (later rebranded as X) during the COVID-19 pandemic to understand key factors associated with vaccine hesitancy by employing deep-learning techniques. Text classification analysis reveals a significant association between attitudes toward vaccination and the unique socio-economic characteristics of US states, such as education, race, income or voting behavior. However, our results indicate that attributing vaccine hesitancy solely to a single social factor is not appropriate. Furthermore, the topic modeling of online discourse identifies two distinct sets of justifications for vaccine hesitancy. The first set pertains to political concerns, including constitutional rights and conspiracy theories. The second pertains to medical concerns about vaccine safety and efficacy. However, vaccine-hesitant social media users pragmatically use broad categories of justification for their beliefs. This behavior may suggest that vaccine hesitancy is influenced by political beliefs, unconscious emotions, and gut-level instinct. Our findings have further implications for the critical role of trust in public institutions in shaping attitudes toward vaccination and the need for tailored communication strategies to restore faith in marginalized communities.