Cold bonding is an energy-efficient method for producing artificial aggregate (AA). However, the relatively high density of AA restricts its use in lightweight concrete products. This study aims to fully upcycle aluminum (Al) content in municipal solid waste incineration bottom ash (IBA) for H2 aeration in the production of lightweight artificial aggregate through alkali activation. To achieve this, the glass particles (GP) in IBA were ground into powder and used as the precursor for NaOH, while the remaining IBA (RIBA), rich in Al, served as the foaming agent. The results showed that RIBA particles (<300 μm) can be used to produce lightweight AA with a loose bulk density ranging from 780 to 840 kg/m3, comparable to sintered fly ash aggregates, and a wide pore size distribution ranging from 0.02 mm to over 1 mm. The inclusion of finer RIBA particles (<75 μm) enhanced the homogeneity of the pore structure, resulting in aggregate strengths of 0.4–0.6 MPa. Moreover, increasing the GP content in IBA up to 40% further boosted the aggregate strength to 1.1 MPa by refining the pore structure and promoting more (C)-N-A-S-H gels with Q4 structures. To ensure sufficient geopolymerization reactions and achieve high strength in the AAs, the alkali concentration should be kept above 3 mol/L. The proposed strategy for lightweight AA production reduced the global warming potential by about 20% compared to the conventional sintered method, offering a more environmentally friendly approach.