A novel property of the quantum critical heavy fermion superconductor $\beta$-YbAlB$_4$ is revealed theoretically. By analyzing the crystalline electronic field (CEF) on the basis of the hybridization picture, odd parity CEF is shown to exist because of sevenfold configuration of B atoms around Yb, which breaks the local inversion symmetry. This allows onsite admixture of 4f and 5d wavefunctions with a pure imaginary coefficient, giving rise to the magnetic toroidal (MT) degree of freedom. By constructing a realistic minimal model for $\beta$-YbAlB$_4$, we show that onsite 4f-5d Coulomb repulsion drives charge transfer between the 4f and 5d states at Yb, which makes the MT fluctuation as well as the electric dipole fluctuation diverge simultaneously with the critical Yb-valence fluctuation at the quantum critical point of the valence transition.