In order to better constrain the alteration history of the Ryugu parent body, we performed a multi-analytical study combining scanning electron microscopy, transmission electron microscopy and infrared spectroscopy on sections extracted from the three fragments A0064-FO019, A0064-FO021 and C0002-FO019 returned from Ryugu by the Hayabusa2 space mission. The three sections show large differences in terms of structure, mineralogy and infrared signature. Section A0064-FO019 resembles the major Ryugu lithology with the presence of both fine-grained phyllosilicates (fg-phyllos) with embedded nanosulfides and coarse-grained phyllosilicates (cg-phyllos), whereas section C0002-FO019 belongs to the group of the less altered lithologies with the presence of anhydrous minerals embedded in a partially amorphous matrix. Section A0064-FO021 also belongs to this group but shows two different lithologies, a compact amorphous one and a more porous and very fractured one showing the presence of Na-rich phosphate, calcite and olivine. The two less altered lithologies (sections A0064-FO021 and C0002-FO019) show the presence of numerous mineralogical features similar to those observed in cometary interplanetary dust particles, ultra-carbonaceous Antarctic micrometeorites or in the CM Paris meteorite, i.e. amorphous and partially crystallized matrix with GEMS-like ghosts objects, whisker olivine, phosphide, or FeNi metal. This supports an outer solar system origin common with that of cometary material for the Ryugu parent body. Combined with the results of Nakamura et al. (2022b) reporting the presence of a lithology showing the presence of GEMS-like objects, we propose that section C0002-FO019 represents the onset of aqueous alteration of such primitive materials. The cg-phyllos and fg-phyllos of section A0064-FO019, i.e. of the major Ryugu lithology, representing the advanced stage of alteration, exhibit distinctive IR signatures with a higher abundance of oxygen-rich functional groups in the organic matter (OM) from the cg-phyllos. We thus suggest the following chronology of formation and evolution for Ryugu: (1) accretion of highly porous aggregate of GEMS-like units with fine-grained high-temperature anhydrous silicates, (2) onset of alteration with the dissolution of primary nanosulfides and development of amorphous/partially crystallized material in the pores, (3) crystallization of fg-phyllos with a second generation of sulfides, (4) later formation of cg-phyllos devoid of nanosulfides and their associated oxygen-rich OM in a more water-rich environment.