We investigate line formation processes in Type IIb supernovae (SNe) from 100 to 500 days post-explosion using spectral synthesis calculations. The modeling identifies the nuclear burning layers and physical mechanisms that produce the major emission lines, and the diagnostic potential of these. We compare the model calculations with data on the three best observed Type IIb SNe to-date - SN 1993J, SN 2008ax, and SN 2011dh. Oxygen nucleosynthesis depends sensitively on the main-sequence mass of the star and modeling of the [O I] 6300, 6364 lines constrains the progenitors of these three SNe to the M_ZAMS=12-16 M_sun range (ejected oxygen masses 0.3-0.9 M_sun), with SN 2011dh towards the lower end and SN 1993J towards the upper end of the range. The high ejecta masses from M_ZAMS >= 17 M_sun progenitors give rise to brighter nebular phase emission lines than observed. Nucleosynthesis analysis thus supports a scenario of low/moderate mass progenitors for Type IIb SNe, and by implication an origin in binary systems. We demonstrate how oxygen and magnesium recombination lines may be combined to diagnose the magnesium mass in the SN ejecta. For SN 2011dh, a magnesium mass of of 0.02-0.14 M_sun is derived, which gives a Mg/O production ratio consistent with the solar value. Nitrogen left in the He envelope from CNO-burning gives strong [N II] 6548, 6583 emission lines that dominate over H-alpha emission in our models. The hydrogen envelopes of Type IIb SNe are too small and dilute to produce any noticeable H-alpha emission or absorption after ~150 days, and nebular phase emission seen around 6550 A is in many cases likely caused by [N II] 6548, 6583. Finally, the influence of radiative transport on the emergent line profiles is investigated...(abridged)