In a companion paper (Keating and Wei 2023 Int. Math. Res. Not. 2024 9607–32), we established asymptotic formulae for the joint moments of higher order derivatives of the characteristic polynomials of CUE random matrices. The leading order coefficients of these asymptotic formulae are expressed as partition sums of derivatives of determinants of Hankel matrices involving I-Bessel functions, with column indices shifted by Young diagrams. In this paper, we continue the study of these joint moments and establish more properties for their leading order coefficients, including structure theorems and recursive relations. We also build a connection to a solution of the σ-Painlevé III ′ equation. In the process, we give recursive formulae for the Taylor coefficients of the Hankel determinants formed from I-Bessel functions that appear at zero and find some differential equations these determinants satisfy. The approach we establish is applicable to determinants of general Hankel matrices whose columns are shifted by Young diagrams.