In this study, we aimed to compare the cusp and crown areas of the maxillary first premolar (PM1), second premolar (PM2), and first molar (M1) in males and females in the Malay population and to formulate sex prediction models. For this purpose, the maxillary posterior teeth of 176 dental cast samples (from 88 males and 88 females) were selected and transformed to two-dimensional digital models using 2D-Hirox KH-7700. Cusp and crown area measurements were obtained using Hirox software by tracing the outermost circumference of the tooth cusps. Statistical analysis included independent t-tests, logistic regression analysis, and receiver-operating characteristic (ROC) curves as well as determination of sensitivity and specificity; analysis was performed with SPSS version 26.0. The significance threshold was set at 0.05. All crown and cusp area measurements were significantly larger in males than in females (p < 0.001). The most sexually dimorphic tooth was the first maxillary molar (mean difference, 10.27 mm2), and the most sexually dimorphic cusp was the mesiopalatal cusp (mean difference, 3.67 mm2) of M1. The sex prediction model had a good accuracy, with 80% of selected cases correctly predicted. Hence, we conclude that the maxillary posterior teeth in the Malay population exhibit significant sexual dimorphism, and this information may be used for sex determination as adjuvants along with other procedures.