Anemia accounts for 8.8% of total disability burden worldwide. Betel quid use among pregnant women has been found to increase anemia risk. Betel quid is prepared by wrapping the betel (or areca) nut, with spices and other additions, in betel or tobacco leaf and it is chewed or placed in the mouth. We explored the association between betel quid use and anemia among men and non-pregnant women. We collected data from a random sample of women and their husbands in Matlab, Bangladesh. Participants reported their current betel quid use and individual characteristics. We assessed hemoglobin (a biomarker of anemia) with a hemoglobinometer and soluble transferrin receptor (a biomarker of iron deficiency) and C-reactive protein (a biomarker of inflammation) in dried blood spots via enzyme immunoassay. We estimated logistic regression models to evaluate the association between betel quid use and anemia and structural equation models (SEM) to evaluate mediating roles of iron deficiency and elevated inflammation. A total of 1133 participants (390 men and 743 non-pregnant women) were included. After controlling for important confounders, any betel quid use was positively associated with anemia among men (OR: 1.80; 95% CI: 1.12, 2.89). Among women, betel quid use was associated with anemia only among the most frequent users (OR: 1.62; 95% CI: 1.03, 2.53). SEM did not reveal indirect paths through inflammation or iron deficiency. Betel quid use may contribute to the burden of anemia among adults in Bangladesh. Our findings suggest the burden of disease attributed to betel quid use has been underestimated.