The hidden symmetry of the asymmetric quantum Rabi model (AQRM) with a half-integral bias (ibQRM$_{\ell}$) was uncovered in recent studies by the explicit construction of operators $J_\ell$ commuting with the Hamiltonian. The existence of such symmetry has been widely believed to cause the degeneration of the spectrum, that is, the crossings on the energy curves. In this paper we propose a conjectural relation between the symmetry and degeneracy for the ibQRM$_{\ell}$ given explicitly in terms of two polynomials appearing independently in the respective investigations. Concretely, one of the polynomials appears as the quotient of the constraint polynomials that assure the existence of degenerate solutions while the other determines a quadratic relation (in general, it defines a curve of hyperelliptic type) between the ibQRM$_{\ell}$ Hamiltonian and its basic commuting operator $J_\ell$. Following this conjecture, we derive several interesting structural insights of the whole spectrum. For instance, the energy curves are naturally shown to lie on a surface determined by the family of hyperelliptic curves by considering the coupling constant as a variable. This geometric picture contains the generalization of the parity decomposition of the symmetric quantum Rabi model. Moreover, it allows us to describe a remarkable approximation of the first $\ell$ energy curves by the zero-section of the corresponding hyperelliptic curve. These investigations naturally lead to a geometric picture of the (hyper-)elliptic surfaces given by the Kodaira-N\'eron type model for a family of curves over the projective line in connection with the energy curves, which may be expected to provide a complex analytic proof of the conjecture.