The objective of this study was to test higher-order statistical (HOS) parameters for the classification of young and elderly voice signals and identify gender- and age-related differences through HOS analysis. This study was based on data from 116 subjects (58 females and 58 males) extracted from the Saarbruecken voice database. In the gender analysis, the same number of voice samples were analyzed for each sex. Further, we conducted experiments on the voices of elderly people using gender analysis. Finally, we reviewed the standards and reference models to reduce sex and gender bias. The acoustic parameters were extracted from young and elderly voice signals using Praat and a time–frequency analysis program (TF32). Additionally, we investigated the gender- and age-related differences in HOS parameters. Young and elderly voice signals significantly differed in normalized skewness (p = 0.005) in women and normalized kurtosis (p = 0.011) in men. Therefore, normalized skewness is a useful parameter for distinguishing between young and elderly female voices, and normalized kurtosis is essential for distinguishing between young and elderly male voices. We will continue to investigate parameters that represent important information in elderly voice signals.