Sensors and wearable technology are used to accumulate health-related data from the visually impaired (VI) and other special persons. This health-related data is disseminated via an open network channel, which poses numerous threats and requires rigorous attention for its protection. An appropriate security framework must be implemented on both the sender and the recipient sides to ensure the privacy and security of sensitive medical information for VI and disabled persons. The telehealth system will then benefit the VI and disabled people by improving their quality of life, telemonitoring, saving time and money, providing online support, and increasing the degree of trust and reliance on the system and the treatment of doctors. These practical benefits are not just theoretical but potentially life-changing for the VI and disabled community. Strong and flawless mutual authentication is necessary for the telehealth system to secure and preserve the privacy of VI and disabled people. Therefore, this article proposes a privacy-preserving asymmetric-based, lightweight, and robust security framework for the telehealth system operationalized for VI and disabled people. The proposed security framework alleviates all threats to sensitive information and preserves the privacy and security of the whole telehealth system. The security of the proposed framework has been verified through a well-known Real-Or-Random (RoR) model and simulated through a ProVerif software verification toolkit, while the performance analysis has been tackled by measuring computation, storage, and communication costs. The comparative analysis result revealed that the proposed framework outperforms its competitors, further reinforcing the hope about its potential impact.