Apple is one of the most popular fruits. In-field grading and sorting of apples would enhance growers’ economic benefits by lowering production costs. This study reviews the key components and progress of the quality inspection algorithm for in-field grading and sorting of apples. Four key components (e.g., conveyor, imaging chamber, sorting actuator, and bin filler) are presented in detail, followed by summarizing the shortcomings of these components. The apple’s external (color, size, and defects) and internal quality inspection technologies, such as optical technologies of visible light, near-infrared (NIR), hyperspectral/multispectral imaging (HSI/MSI), and structured illumination (SI) were presented. Despite the excellent detection performance of emerging technologies (e.g., HSI, MSI, and SI), visible light is still dominantly used for in-filed grading. Challenges in getting information on the whole surface area of the apple, uneven lighting, machine size, throughput, and associated costs hamper the commercialization of apple in-field grading and sorting equipment. At present, more efforts should be devoted to internal quality inspection, by developing reliable, fast, and accurate detection equipment and algorithms. With the advancement of sensors and automation algorithms, as well as the emergence of mechanical systems that are suitable for in-field use, it is anticipated that the apple in-field grading and sorting equipment that inspects both external and internal quality will be realized and commercialized in the near future.