We explore the quantum criticality of a two-site model combining quantum Rabi models with hopping interaction. Through a combination of analytical and numerical approaches, we find that the model allows the appearance of a superradiant quantum phase transition (QPT) even in the presence of strong $\mathbf{A}^2$ terms, preventing single-site superradiance. In the two-site model the effect of $\mathbf{A}^2$ terms can be surmounted by the photon delocalization from hopping, and a reversed superradiant QPT occurs as a consequence of the competition between $\mathbf{A}^2$ terms and the hopping interaction. We characterize the phase diagram and scaling functions, and extract the critical exponents in the vicinity of the critical point, thus establishing the universal behavior of the second-order phase transition. Remarkably the effective hopping strength will be enhanced if more cavities are cascaded. We also prove that the multi-qubit counterpart of the quantum Rabi dimer, i.e., the Dicke dimer, has the same properties in beating the $\mathbf{A}^2$ effect. Our work provides a way to the study of phase transitions in presence of the $\mathbf{A}^2$ terms and offers the prospect of investigating quantum-criticality physics and quantum devices in many-body systems.