Life cycle inventories (LCIs) and life cycle assessments (LCAs) of photovoltaic (PV) modules and their components focus on the operations of PV factories, but the factories and industrial site product and construction stages are either not or only partially tackled. This work contributes through the bottom-up, model-based generation of LCIs and LCAs for setting up a vertically integrated 5 GWp/a PV industrial site, including the manufacturing of silicon ingots, wafers, solar cells, and PV modules, on a 50 ha greenfield location. Two comparative LCAs are performed. The first compares the annualized environmental impacts of the developed LCI sets with four existing inventories in the Ecoinvent v3.8 database. The second comparative LCA explores the environmental impact differences concerning the industrial site when using different building systems for the factories. Here, the reference system with a steel structure is compared with two alternative building systems: precast concrete and structural timber. The results show that the wafer, cell, and module factories’ annualized environmental impacts with the Ecoinvent LCIs are strongly overestimated. For the ingot factory, the opposite result is identified. The impacts of all four factories show reductions of between 11.7% and 94.3% for 14 of the 15 impact categories. High mean environmental impact shares of 79.0%, 78.2% and 79.2% for the steel, precast concrete and timber structural building systems, respectively, are generated at the product stage. The process and facilities equipment generates 54.2%, 54.4% and 58.2% of the total product and construction stages’ mean environmental impact shares. The proposed alternative timber building system reduces the environmental impacts in 14 of the 15 evaluated categories, with reductions ranging from 1.1% to 12.4%.