The use and sale of sexual enhancement drugs (particularly unapproved aphrodisiacs) have become a public health concern in Ghana and many other sub-Saharan African countries. While most studies have examined this phenomenon from the level of individual perspectives, this study investigates the multi-dimensional and multi-level factors (e.g., individual characteristics and behaviours, interpersonal factors, community norms and practices, institutional and public policy factors) that influence attitudes, perceptions, and use of aphrodisiacs among men and women in Ghana. Using a concurrent mixed-method design, we derived the data from a semi-structured interview and cross-sectional survey conducted across five administrative regions in Ghana. Interpretative phenomenological analysis and logistic regression techniques were used to analyse the qualitative and quantitative (survey) data, respectively. Approximately 12.6% of participants (17.6% among males and 7.2% among females) had used an aphrodisiac in the six months prior to the study. Approximately 23.4% of the participants had more than one partner during the same period. Among men, being religious (B = −0.238, p < 0.05) and having multiple sexual partners (B = 0.481, p < 0.01) were positively associated with the use of aphrodisiacs. For women, being employed (B = −1.539, p < 0.01), engaging in physical activities (exercising) (B = −0.658, p < 0.05), having good health (B = 0.869, p < 0.05), having multiple sexual partners (B = 1.191, p < 0.01), and taking alcohol (B = 1.041, p < 0.01) were associated with use of aphrodisiacs. Although many participants had used aphrodisiacs, women, in particular, held unfavourable views about the drugs due to perceived negative health implications for themselves and their partners. The findings also show that community-level factors (e.g., social norms and expectations), interpersonal factors (e.g., expectations of partners and friends), public policy (e.g., drug-related regulations), and organisational/institutional factors (e.g., health system arrangements about access and use of drugs) were critical to the sale and use of aphrodisiacs among both men and women in Ghana. A multi-level analysis of the use of sexual enhancement drugs among men and women is crucial to formulating social and public health policies that aim to improve public knowledge of these drugs, reduce uncontrolled production, and protect population health and well-being.