The adoption of electric vehicles (EVs) is transforming the landscape of energy consumption. While the technical and economic dimensions of EV adoption are increasingly well understood, the aspect of justice in demand flexibility remains underexplored. This study examines the complex relationship between flexibility in EV charging behaviors and the influence of socio-psychological and justice factors. We explore a range of demographic and social-psychological variables including charging anxiety, environmental concerns, perceived cost-saving perception, perceived privacy, and trust in utility providers. Our results reveal that these variables positively influence the changes in charging habits, including time-shifting and load-reduction. This study also uncovers disparities in charging behavior adjustments across various demographics groups. For instance, White respondents are more likely to charge their EVs during off-peak hours than their non-White counterparts and homeowners show a greater intention to reduce EV charging load during peak hours compared to renters. Additionally, high-income individuals exhibit a stronger willingness to shift charging times to off-peak, with White respondents within the high-income group being the most likely to reduce the amount of charging load during peak hours. Conversely, low-income White respondents are less inclined to make such adjustments. These disparities are likely tied to socioeconomic status, as more vulnerable groups often face greater constraints in adjusting their schedules. Therefore, it is imperative that policies prioritize flexibility justice by addressing the specific needs and behaviors of vulnerable groups, aiming to mitigate the additional burdens resulting from their limited flexibility.