Hecke Triangle Groups, Transfer Operators and Hausdorff Dimension

Abstract

We consider the family of Hecke triangle groups Gamma _{w} = langle S, T_wrangle generated by the Möbius transformations S : zmapsto -1/z and T_{w} : z mapsto z+w with w > 2. In this case, the corresponding hyperbolic quotient Gamma _{w}backslash {mathbb {H}}^2 is an infinite-area orbifold. Moreover, the limit set of Gamma _w is a Cantor-like fractal whose Hausdorff dimension we denote by delta (w). The first result of this paper asserts that the twisted Selberg zeta function Z_{Gamma _{ w}}(s, rho ) , where rho : Gamma _{w} rightarrow mathrm {U}(V) is an arbitrary finite-dimensional unitary representation, can be realized as the Fredholm determinant of a Mayer-type transfer operator. This result has a number of applications. We study the distribution of the zeros in the half-plane mathrm {Re}(s) > frac{1}{2} of the Selberg zeta function of a special family of subgroups ( Gamma _w^N )_{Nin {mathbb {N}}} of Gamma _w. These zeros correspond to the eigenvalues of the Laplacian on the associated hyperbolic surfaces X_w^N = Gamma _w^N backslash {mathbb {H}}^2. We show that the classical Selberg zeta function Z_{Gamma _w}(s) can be approximated by determinants of finite matrices whose entries are explicitly given in terms of the Riemann zeta function. Moreover, we prove an asymptotic expansion for the Hausdorff dimension delta (w) as wrightarrow infty .