Assessing physical frailty (PF) is vital for early risk detection, tailored interventions, preventive care, and efficient healthcare planning. However, traditional PF assessments are often impractical, requiring clinic visits and significant resources. We introduce a video-based frailty meter (vFM) that utilizes machine learning (ML) to assess PF indicators from a 20s exercise, facilitating remote and efficient healthcare planning. This study validates the vFM against a sensor-based frailty meter (sFM) through elbow flexion and extension exercises recorded via webcam and video conferencing app. We developed the vFM using Google's MediaPipe ML model to track elbow motion during a 20s elbow flexion and extension exercise, recorded via a standard webcam. To validate vFM, 65 participants aged 20-85 performed the exercise under single-task and dual-task conditions, the latter including counting backward from a random two-digit number. We analyzed elbow angular velocity to extract frailty indicators-slowness, weakness, rigidity, exhaustion, and unsteadiness-and compared these with sFM results using intraclass correlation coefficient analysis and Bland-Altman plots. The vFM results demonstrated high precision (0.00-7.14%) and low bias (0.00-0.09%), showing excellent agreement with sFM outcomes (ICC(2,1): 0.973-0.999), unaffected by clothing color or environmental factors. The vFM offers a quick, accurate method for remote PF assessment, surpassing previous video-based frailty assessments in accuracy and environmental robustness, particularly in estimating elbow motion as a surrogate for the 'rigidity' phenotype. This innovation simplifies PF assessments for telehealth applications, promising advancements in preventive care and healthcare planning without the need for sensors or specialized infrastructure.