The use of automatic systems in the agriculture sector enhances product quality and the country’s economy. The method used to sort fruits and vegetables has a remarkable impact on the export market and quality assessment. Although manual sorting and grading can be performed easily, it is inconsistent, time-consuming, expensive, and highly influenced by the surrounding environment. In this regard, this study aimed to design and optimize the performance of a low-cost, multi-purpose, automatic RGB color-based sensor for sorting fruits. The proposed automatic color sorting system consists of hardware components including a machine frame, belt and pulleys, conveyor belt, scanning zone, plastic boxes, electric components (stepper motors, RGB color sensors, Arduino Mega, motor drivers), and software components (Arduino IDE version 2.2.1 and C++). Calibration was performed for the light intensity sensor to measure the light intensity inside the scanning zone, the conveyor speed sensor, and the RGB color sensors by testing the RGB color channels. The sensor, the height, conveyor belt color, and light intensity should be carefully adjusted to ensure a high performance of the color-based sorting system. The results showed that the appropriate sensor height ranged from 15 to 30 mm, the optimum color of the conveyor belt was black, and scanning the objects at a light intensity of 25 lux achieved the best output signals. The RGB color sensors achieved an analytical performance similar to that obtained with manual sorting without requiring the use of computers for image processing like other automatic sorting systems do in order to gather RGB data.