In Uganda, informal raw milk sales dominate for domestic dairy consumption. This study was implemented to identify the structure of the dairy value chain starting from farms that participated in the Japan International Cooperation Agency Safe Milk Promotion in Mbarara project conducted between 2016 and 2019, to assess the hygiene conditions along the chain, and thereby identify the bottleneck of dairy hygiene intervention. A longitudinal study was conducted in 30 dairy farms in Mbarara District to compare the practice, prevalence of sub-clinical mastitis, and level of milk hygiene in 2016–2017 and 2019, before and after the milking hygiene intervention in 2018. California Mastitis Test was used for diagnosis with sub-clinical mastitis. Bulk milk samples were collected and a checklist was used to examine hygiene practices by observation. A cross-sectional study was conducted in 15 milk collecting centers using a structured questionnaire to quantify the dairy value chain, and to sample milk from cooler tanks in 2020. Microbiological examinations of bulk milk from farms and collection centers were conducted using six-point blood agar scoring and 3M Petri film, respectively. Participatory online appraisals with farmers and dairy cooperatives union were conducted to better understand the overall dairy value chains. The cooperatives sold milk to both formal and informal chains, but the sale of raw milk to Kampala was conducted by independent private traders. Within-herd prevalence of sub-clinical mastitis significantly decreased from 72.3% before the intervention to 25.8% after (p < 0.001). However, the farm bulk milk score did not change (3.3 vs. 3.2, p = 0.418). A significant increase in the total bacterial count was observed in the milk from collection centers (mean: 6.50 log10 CFU/ml) when compared to farm bulk milk (mean: 3.79 log10 CFU/ml; p < 0.001). Only 13.3% of the samples from the centers met the microbiological criteria for processing for human consumption. Our findings suggest that intervention targeted only at mastitis does not lead to better public health due to the low level of hygiene in transportation and milk handling in milk collection centers. Systematic interventions are needed to improve post-harvest dairy hygiene in Uganda.