Chimpanzees form a multi-male/multi-female unit-group and the members of the group usually split into temporary parties. They sometimes exchange long-distance calls, pant-hoots (PH) between parties and utter PH in chorus within a party. Although most studies on PH have focused on the sociobiological functions for vocalizers or the referential functions, such studies based on individualistic or anthropocentric viewpoint do not tell us how chimpanzees interact with others by using PH. This study aims to analyze how the chimpanzees of Mahale connect their actions through PH and how they organize their social fields beyond visual contact with involvement of others in sight, by applying ethnomethodology and focusing on hearers’ actions. Case analyses based on the result of their usual interval for exchanging PH (10 sec.) revealed that both vocalizers and hearers practiced the same “call-answer” form. Utilizing this form, not only vocalizers could construct an auditory social field between parties but also non-answering hearers could observe the field. Even if PH hearers headed to the place from which they heard the PH, they sometimes restrained themselves from answering to hear whether or not the answer was uttered by other parties, and anew uttered PH to elicit the voluntary answer from the party with whom they tried to interact. Meanwhile, PH hearers sometimes answered immediately regardless of the contexts in which the PH was uttered. Employing these two hands, they would organize their auditory social fields. Once two parties constructed their auditory social field by exchanging PH, one could wait for next call from another or they could get to meet visually with repeating PH exchanges. On the other hand, PH hearers sometimes terminated or deferred those interactions, which hearers’ attitudes were supposed to generate the society in which the members could stay apart from each other even beyond PH distance. In conclusion, these hearers’ various actions show the sociality related to their dynamic fission-fusion society and engaging in non-referential interactions. When we say that chimpanzees’ society is complex, we may be often thinking of the process of interactions resulted from such their unique sociality.