The traditional pan-tilt-zoom (PTZ) camera relies on manual operation in control, and it requires a large number of samples for training in the algorithm to achieve higher target recognition accuracy. Therefore, we introduce the Boosted Efficient Binary Local Image Descriptor (BEBLID) propose a pedestrian detection and tracking method applied to the PTZ. By extracting image feature points and feature descriptions, we use Support Vector Machines (SVM) to train fewer samples to complete pedestrian detection. Then we input it into kernel correlation filters (KCF) for subsequent tracking, and use its high frame rate to reduce the feedback steps of the PTZ position loop, which reduces the system running time, and then we control the target image to be always locked in the center of the image. Finally, the experiment is carried out, and the whole system can run normally. Under 700 test samples, the detection effect of this algorithm is the highest, reaching 93.00 %, which is 14.57 % and 3.57 % higher than that of Scale-invariant feature transform (SIFT) and Oriented Brief (ORB).