The GaAs1–xBix semiconductor alloy allows one to achieve large bandgap reduction and enhanced spin–orbit splitting energy at dilute Bi quantities. The bismide is currently being developed for near- to mid-infrared lasers, multi-junction solar cells, and photodetectors. In this structure–property relationship study of GaAsBi alloys, we report polarization dependent photoluminescence that reaches a polarization ratio up to 2.4 at room temperature. Polarization dependence is also presented using transmittance spectra, birefringence, and linear dichroism. The optical anisotropy observations agree with the predictions of point symmetry reduction in the CuPtB-type ordered GaAsBi phase. The structural ordering is investigated experimentally from the atomic scale in molecular-beam epitaxy (MBE) grown samples on exact and miscut (001) GaAs substrates, as well as on (001) Ge. The latter sample is composed of anti-phase domains in which the ordering axes are rotated by 90° angles. Since the conditions stabilizing the CuPtB ordered phase fall within the typical MBE growth regime of dilute bismides, the optical anisotropy in GaAsBi alloys is expected to be ubiquitous. These findings are important for the future development of GaAsBi-based optoelectronics and also provide new means to analyze structurally complex bismide alloys.