Infection duration affects individual host fitness and between-host transmission. Whether an infection is cleared or becomes chronic depends on the complex interaction between host immune responses and parasite growth. Empirical and theoretical studies have suggested that there are critical thresholds of parasite dose that can determine clearance versus chronicity, driven by the ability of the parasite to manipulate host immunity. However, the mammalian immune response is characterized by strong positive and negative feedback loops that could generate duration thresholds even in the absence of direct immunomodulation. Here, we derive and analyse a simple model for the interaction between T-cell subpopulations and parasite growth. We show that whether an infection is cleared or not is very sensitive to the initial immune state, parasite dose and strength of immunological feedbacks. In particular, chronic infections are possible even when parasites provoke a strong and effective immune response and lack any ability to immunomodulate. Our findings indicate that the initial immune state, which often goes unmeasured in empirical studies, is a critical determinant of infection duration. This work also has implications for epidemiological models, as it implies that infection duration will be highly variable among individuals, and dependent on each individual's infection history.