First principles investigations of the high temperature superconducting system Ba$_2$CuO$_{3+\delta}$, recently discovered at $\delta\approx0.2$ at $T_c=70$ K, are applied to demonstrate the effects of oxygen ordering on the electronic and magnetic properties. The observed `highly over-doped' superconducting phase displays stretched Cu-planar oxygen O$_{\rm P}$ distances and anomalously shortened Cu-apical O$_{\rm A}$ separations compared with other cuprates. The stoichiometric system $\delta=0$, with its strongly one-dimensional (1D) Cu-O$_{\rm P}$ chain structure, when nonmagnetic shows 1D Fermi surfaces that lead, within density functional theory, to antiferromagnetic Cu-O$_{\rm P}$ chains (a spin-Peierls instability). Accounting for 1D fluctuations and small interchain coupling according to the theory of Schulz indicates this system, like Sr$_2$CuO$_3$, is near the 1D Luttinger-liquid quantum critical phase. The unusual Cu-O bond lengths per se have limited effects on other properties for $\delta$=0. We find that a `doubled bilayer' structure of alternating Cu-O$_{\rm P}$ chains and wide rung Cu$_3$O$_4$ ladders is the energetically preferred one of three possibilities where the additional oxygen ions bridge Cu-O$_{\rm P}$ chains in the superconducting phase $\delta=1/4$. Nominal formal valences of the three Cu sites are discussed. The six-fold (octahedral) site is the most highly oxidized, accepting somewhat more holes in the $d_{z^2}$ orbital than in the $d_{x^2-y^2}$ orbital. The implication is that two-band physics is involved in the pairing mechanism and the superconducting carriers. The Fermi surfaces of this metallic bilayer structure show both 1D and 2D strong (incipient) nesting instabilities, possibly accounting for the lack of clean single-phase samples based on this structure and suggesting importance for the pairing mechanism.