We construct black hole solutions in four-dimensional quadratic gravity, supported by a scalar field conformally coupled to quadratic terms in the curvature. The conformal matter Lagrangian is constructed with powers of traces of a conformally covariant tensor, which is defined in terms of the metric and a scalar field, and has the symmetries of the Riemann tensor. We find exact, neutral and charged, topological black hole solutions of this theory when the Weyl squared term is absent from the action functional. Including terms beyond quadratic order on the conformally covariant tensor, allows to have asymptotically de Sitter solutions, with a potential that is bounded from below. For generic values of the couplings we also show that static black hole solutions must have a constant Ricci scalar, and provide an analysis of the possible asymptotic behavior of both, the metric as well as the scalar field in the asymptotically AdS case, when the solutions match those of general relativity in vacuum at infinity. In this frame, the spacetime fulfils standard asymptotically AdS boundary conditions, and in spite of the non-standard couplings between the curvature and the scalar field, there is a family of black hole solutions in AdS that can be interpreted as localized objects. We also provide further comments on the extension of these results to higher dimensions.