The connection problem for orthogonal polynomials is, given a polynomial expressed in the basis of one set of orthogonal polynomials, computing the coefficients with respect to a different set of orthogonal polynomials. Expansions in terms of orthogonal polynomials are very common in many applications. While the connection problem may be solved by directly computing the change–of–basis matrix, this approach is computationally expensive. A recent approach to solving the connection problem involves the use of the spectral connection matrix, which is a matrix whose eigenvector matrix is the desired change–of–basis matrix. In Bella and Reis (2014), it is shown that for the connection problem between any two different classical real orthogonal polynomials of the Hermite, Laguerre, and Gegenbauer families, the related spectral connection matrix has quasiseparable structure. This result is limited to the case where both the source and target families are one of the Hermite, Laguerre, or Gegenbauer families, which are each defined by at most a single parameter. In particular, this excludes the large and common class of Jacobi polynomials, defined by two parameters, both as a source and as a target family. In this paper, we continue the study of the spectral connection matrix for connections between real orthogonal polynomial families. In particular, for the connection problem between any two families of the Hermite, Laguerre, or Jacobi type (including Chebyshev, Legendre, and Gegenbauer), we prove that the spectral connection matrix has quasiseparable structure. In addition, our results also show the quasiseparable structure of the spectral connection matrix from the Bessel polynomials, which are orthogonal on the unit circle, to any of the Hermite, Laguerre, and Jacobi types. Additionally, the generators of the spectral connection matrix are provided explicitly for each of these cases, allowing a fast algorithm to be implemented following that in Bella and Reis (2014).