We introduce an invariant of a hyperbolic knot which is a map $\alpha\mapsto \boldsymbol{\Phi}_\alpha(h)$ from $\mathbb{Q}/\mathbb{Z}$ to matrices with entries in $\overline{\mathbb{Q}}[[h]]$ and with rows and columns indexed by the boundary parabolic ${\rm SL}_2(\mathbb{C})$ representations of the fundamental group of the knot. These matrix invariants have a rich structure: (a) their $(\sigma_0,\sigma_1)$ entry, where $\sigma_0$ is the trivial and $\sigma_1$ the geometric representation, is the power series expansion of the Kashaev invariant of the knot around the root of unity ${\rm e}^{2\pi{\rm i} \alpha}$ as an element of the Habiro ring, and the remaining entries belong to generalized Habiro rings of number fields; (b) the first column is given by the perturbative power series of Dimofte-Garoufalidis; (c) the columns of $\boldsymbol{\Phi}$ are fundamental solutions of a linear $q$-difference equation; (d) the matrix defines an ${\rm SL}_2(\mathbb{Z})$-cocycle $W_\gamma$ in matrix-valued functions on $\mathbb{Q}$ that conjecturally extends to a smooth function on $\mathbb{R}$ and even to holomorphic functions on suitable complex cut planes, lifting the factorially divergent series $\boldsymbol{\Phi}(h)$ to actual functions. The two invariants $\boldsymbol{\Phi}$ and $W_\gamma$ are related by a refined quantum modularity conjecture which we illustrate in detail for the three simplest hyperbolic knots, the $4_1$, $5_2$ and $(-2,3,7)$ pretzel knots. This paper has two sequels, one giving a different realization of our invariant as a matrix of convergent $q$-series with integer coefficients and the other studying its Habiro-like arithmetic properties in more depth.