We use an atomistic model to consider the effect of shape symmetry breaking on the optical properties of self-assembled InAs/GaAs quantum dots. In particular, we investigate the energy level structure and optical activity of the lowest energy excitons in these nanostructures. We compare between quantum dots with twofold rotational and two reflections $({C}_{2v})$ symmetry and quantum dots in which this symmetry was reduced to one reflection only $({C}_{s})$ by introducing a facet between the quantum dots and the host material. We show that the symmetry reduction mostly affects the optical activity of the dark exciton. While in symmetric quantum dots, one of the dark exciton eigenstates has a small dipole moment polarized along the symmetry axis (growth direction) of the quantum dot, in nonsymmetric ones, the two dark excitons' dipole moments are predominantly cross-linearly polarized perpendicular to the growth direction and reveal pronounced polarization anisotropy. Our model calculations agree quantitatively with recently obtained experimental data.