Background: This study aimed to survey the attitudes toward COVID-19 vaccines and their acceptability among the Japanese public as soon as the United States Food and Drug Administration (FDA) authorized vaccines and their rollouts started around the world. Methods: An anonymous cross-sectional survey was conducted in Japan between 4 January and 5 March 2021. A questionnaire was administered to evaluate attitudes toward COVID-19 vaccines according to demographic characteristics, vaccine characteristics, and vaccine production. Results: A total of 1037 completed responses were received. More than half (63.5%) of the participants responded positively (extremely likely/likely) toward COVID-19 vaccines. The highest acceptance to be vaccinated was discovered among the youngest age group. As expected, participants who had never delayed acceptance or refused the vaccine in their history of vaccination had a significantly higher willingness to be vaccinated against COVID-19 (p < 0.001). Females (OR = 2.66, 95% CI: 1.99−3.58) and participants who had ever delayed acceptance or refuse the vaccine (OR = 3.49, 95% CI: 2.42−5.05) had higher odds of COVID-19 vaccine hesitancy. Participants with a postgraduate degree (OR = 0.64, 95% CI: 0.40−1.00) presented the highest willingness to be vaccinated against COVID-19. More than two-thirds (72.9%, 95% CI: 70.4%−75.8%) of the participants did not mind a booster dose required following primary vaccination. A total of 63.2% (95% CI: 60.0%−66.0%) of the participants only accepted a nearly 90% effective or above vaccine at preventing COVID-19. At the same, 86.4% (95% CI: 84.4%−88.4%) of the participants reported only accepting a vaccine with minor side effects. Conclusions: The moderate levels of COVID-19 vaccine acceptance found in the early phase of the pandemic demonstrate that it is important to improve the implementation of effective management for vaccine promotion and the acceptability of the vaccine to slow or delay transmission.