There are now more violent conflicts globally than at any time in the past three decades, resulting in the largest forced displacement crisis ever recorded. Understanding at a granular level the well-being of refugees is essential to inform successful poverty alleviation strategies and unlock refugees’ potential. As forced displacement can lead to a reorganization of a family’s structure, we use a structural model in combination with data from refugee camps and surrounding communities in Uganda and Kenya to estimate the allocation of consumption within families. We compute poverty rates that account for intra-household inequality, finding that refugee children can be up to three times more likely to be poor than adults. So, refugee children not only suffer from the experience of forced migration, but also from potentially low nutrition and a disproportionately higher poverty risk. Using a supervised machine learning algorithm, we show that a small set of observable traits, such as a child’s age, household composition, and access to sanitation and clean water, predict child poverty in refugee settlements and surrounding communities remarkably well, often better than per-capita household expenditure.