Vaccines stand out as one of the most effective tools in our arsenal for reducing morbidity and mortality. Nonetheless, public hesitancy towards vaccination often stems from concerns about potential side effects, which can vary from person to person. As of now, there are no automated systems available to proactively warn against potential side effects or gauge their severity following vaccination. We have developed machine learning (ML) models designed to predict and detect the severity of post-vaccination side effects. Our study involved 2111 participants who had received at least one dose of either a COVID-19 or influenza vaccine. Each participant was equipped with a Garmin Vivosmart 4 smartwatch and was required to complete a daily self-reported questionnaire regarding local and systemic reactions through a dedicated mobile application. Our XGBoost models yielded an area under the receiver operating characteristic curve (AUROC) of 0.69 and 0.74 in predicting and detecting moderate to severe side effects, respectively. These predictions were primarily based on variables such as vaccine type (influenza vs. COVID-19), the individual's history of side effects from previous vaccines, and specific data collected from the smartwatches prior to vaccine administration, including resting heart rate, heart rate, and heart rate variability. In conclusion, our findings suggest that wearable devices can provide an objective and continuous method for predicting and monitoring moderate to severe vaccine side effects. This technology has the potential to improve clinical trials by automating the classification of vaccine severity.