Context: CoRoT-2b is one of the most anomalously large exoplanet known. Given its large mass, its large radius cannot be explained by standard evolution models. Interestingly, the planet's parent star is an active, rapidly rotating solar-like star with a large fraction (7 to 20%) of spots. Aims: We want to provide constraints on the properties of the star-planet system and understand whether the planet's inferred large size may be due to a systematic error on the inferred parameters, and if not, how it may be explained. Methods: We combine stellar and planetary evolution codes based on all available spectroscopic and photometric data to obtain self-consistent constraints on the system parameters. Results: We find no systematic error in the stellar modeling (including spots and stellar activity) that would yield the required ~10% reduction in size for the star and thus the planet. Two classes of solutions are found: the usual main sequence solution for the star yields for the planet a mass of 3.67+/-0.13 Mjup, a radius of 1.55+/-0.03 Rjup for an age that is at least 130Ma, and should be less than 500Ma given the star's fast rotation and significant activity. We identify another class of solutions on the pre-main sequence, in which case the planet's mass is 3.45\pm 0.27 Mjup, its radius is 1.50+/-0.06 Rjup for an age between 30 and 40 Ma. These extremely young solutions provide the simplest explanation for the planet's size which can then be matched by a simple contraction from an initially hot, expanded state, provided the atmospheric opacities are increased by a factor ~3 compared to usual assumptions for solar compositions atmospheres. Other solutions imply in any case that the present inflated radius of CoRoT-2b is transient and the result of an event that occurred less than 20 Ma ago: a giant impact with another Jupiter-mass planet, or interactions with another object in the system which caused a significant rise of the eccentricity followed by the rapid circularization of its orbit. Conclusions: Additional observations of CoRoT-2 that could help understanding this system include searches for infrared excess and the presence of a debris disk and searches for additional companions. The determination of a complete infrared lightcurve including both the primary and secondary transits would also be extremely valuable to constrain the planet's atmospheric properties and to determine the planet-to-star radius ratio in a manner less vulnerable to systematic errors due to stellar activity.