In this study, we designed a rapid colorimetric sensing system using competitive ELISA test on a microfluidic paper-based analytical device for the detection of ketamine, a frequently abused drug. Oral fluid was selected for the test to facilitate the collection of samples based on its advantages of low infection risk, noninvasiveness of sample collection, and decreased chance of sample adulteration. After optimization of the operation parameters, including reaction temperature, vibration washing time, reaction time, and antibody concentration, the resulting assay can be completed in as little as 6 min with a detection limit of 0.03 ng/mL. Moreover, we developed an Android smartphone app to analyze and measure the results of the test, further enhancing the test’s portability, and image recording and data transmission capabilities. In order to test the feasibility and performance of the optimized colorimetric assay, 90 oral fluid samples from drug abuse patients were tested. The paper-based platform features 90% sensitivity (confidence interval (CI): 76.34–97.21%) and 92% specificity (CI: 80.77–97.78%) for ketamine analysis. This competitive paper-based ELISA sensing system provides a rapid, convenient, sensitive, and high-throughput approach for drug monitoring.