We study the stability of ($4+1$)-dimensional charged Gauss-Bonnet black holes and solitons. We observe an instability related to the condensation of a scalar field and construct explicit ``hairy'' black hole and soliton solutions of the full system of coupled field equations. We investigate the cases of a massless scalar field as well as that of a tachyonic scalar field. The solitons with scalar hair exist for a particular range of the charge and the gauge coupling. This range is such that for intermediate values of the gauge coupling a ``forbidden band'' of charges for the hairy solitons exists. We also discuss the behavior of the black holes with scalar hair when changing the horizon radius and/or the gauge coupling and find that various scenarios at the approach of a limiting solution appear. One observation is that hairy Gauss-Bonnet black holes never tend to a regular soliton solution in the limit of vanishing horizon radius. We also prove that extremal Gauss-Bonnet black holes can not carry massless or tachyonic scalar hair and show that our solutions tend to their planar counterparts for large charges.