Online systems face a major challenge in efficiently monitoring participants and students throughout lectures, particularly during exams. Establishing robust approaches and technologies is essential for identifying unjust, unethical, and illegal conduct in educational environments, particularly during academic courses and examinations. This study introduces a novel online proctoring system that employs deep learning to continuously supervise physical locations without requiring a human proctor. The system utilizes face detection and recognition algorithms to implement biometric procedures, including facial recognition. This work presents a novel approach for face recognition training, which involves an incremental training procedure. This eliminates the requirement for an additional step, resulting in reduced computation cost and time. In order to achieve high accuracy, the suggested model evaluated three distinct face detectors: HOG, MTCNN, and Yoloface. The assessment of the suggested model demonstrates that the HOG approach has surpassed the others. The proposed approach yields a significantly high accuracy rate.