Secure multi-party computation is an important research area in cryptography, and the secret sharing scheme (SSS) is one of the main tools for constructing multi-party computation protocols. The access structure and the adversary structure are two important subsets of participants in an SSS. In general, the collection of all qualified subsets that can reconstruct the secret s, is known as an access structure, while no information regarding this secret is available to any unqualified subsets, and the collection of unqualified subsets is described as an adversary structure. The maximal adversary, which will become a qualified subset if any one participant not in this unqualified subset is added. At present, there is no effective algorithm to determine the maximal adversary structure for any given access structure. In this paper, we propose two algorithms to determine the maximal adversary structure from any given access structure, in which a binary tree is introduced to construct such algorithms. Moreover, a special type of access structure is established, from which the maximal adversary structure can be directly characterized, and the maximal adversary structure in this case is shown to be the largest when the number of participants of each qualified polynomial in the access structure is three.