In this work, we investigate the influence of the molecular beam epitaxy (MBE) growth conditions (substrate temperature and arsenic flux) on the photovoltaic (PV) behavior and asymmetric characteristics of nominally identical well-doped AlGaAs/AlAs/GaAs double-barrier quantum well infrared photodetectors. This PV effect, already studied and reported in the literature, has been attributed to unintentional asymmetries of the potential profile introduced during the MBE growth process; in particular, due to an inequivalence of the AlAs layer properties or, more plausibly, to local space-charge regions originating from silicon segregation. The different “unintended” asymmetries for the samples considered in this work, validated by both dark-current and responsivity measurements, point at first glance to the existence of structural dissimilarities affecting the PV response. Hence, in order to clarify the influence of the suggested AlAs barriers inequivalence or interface roughness and quality in the origin of the PV signal we have performed a direct layer structural characterization by cross-section high resolution transmission electron microscopy. The analysis yields that regardless of the different growth conditions, the layers properties are similar, suggesting they play a minor role in the origin of the PV effect. Also this characterization tool may provide a further evidence of Si segregation being the main responsible. Concerning its growth conditions dependence, it seems that the As flux, and not only the substrate temperature, may affect Si segregation and hence the PV response.