Holomorphic functions play a crucial role in operator theory and the Cauchy formula is a very important tool to define the functions of operators. The Fueter–Sce–Qian extension theorem is a two-step procedure to extend holomorphic functions to the hyperholomorphic setting. The first step gives the class of slice hyperholomorphic functions; their Cauchy formula allows to define the so-called S-functional calculus for noncommuting operators based on the S-spectrum. In the second step this extension procedure generates monogenic functions; the related monogenic functional calculus, based on the monogenic spectrum, contains the Weyl functional calculus as a particular case. In this paper we show that the extension operator from slice hyperholomorphic functions to monogenic functions admits various possible factorizations that induce different function spaces. The integral representations in such spaces allow to define the associated functional calculi based on the S-spectrum. The function spaces and the associated functional calculi define the so-called fine structure of the spectral theories on the S-spectrum. Among the possible fine structures there are the harmonic and polyharmonic functions and the associated harmonic and polyharmonic functional calculi. The study of the fine structures depends on the dimension considered and in this paper we study in detail the case of dimension five, and we describe all of them. The five-dimensional case is of crucial importance because it allows to determine almost all the function spaces will also appear in dimension greater than five, but with different orders.