We consider the question: Can the isotropy representation of an irreducible pseudo-Riemannian symmetric space be realized as a conformal holonomy group? Using recent results by Čap, Gover and Hammerl, we study the representations of SO(2,1), PSU(2,1) and PSp(2,1) as isotropy groups of irreducible symmetric spaces of signature (3,2), (4,4) and (6,8), respectively, describing the geometry induced by a conformal holonomy reduction to the corresponding subgroups. In the case of SO(2,1), we show that conformal manifolds with such a holonomy reduction are always locally conformally flat and hence this group cannot be a conformal holonomy group. This result completes the classification of irreducible conformal holonomy groups in Lorentzian signature. In the case of PSU(2,1), we show that conformal manifolds of signature (3,3) with this holonomy reduction carry, on an open dense subset, a canonical nearly para-Kähler metric with positive Einstein constant. For PSp(2,1) we also show that there is an open dense subset endowed with a canonical Einstein metric in the conformal class. As a result, after restricting to an open dense subset, the conformal holonomy must be a proper subgroup of PSU(2,1) or of PSp(2,1), respectively. These are special cases of an interesting relationship between a class of special conformal holonomy groups, and non-integrable geometries with skew symmetric, parallel torsion, which we also explore. Finally, using a recent result of Graham and Willse we prove the following general non-existence result: for a real-analytic, odd-dimensional conformal manifold, the conformal holonomy group can never be given by the isotropy representation of an irreducible pseudo-Riemannian symmetric space unless the isotropy group is SO0(p+1,q+1).