Many low- and middle-income countries have adopted telemedicine programs that connect frontline health workers (FHWs) such as nurses, midwives, or community health workers in rural and remote areas with physicians in urban areas to deliver care to patients. By leveraging technology to reduce temporal, financial, and geographical barriers, these health worker-to-physician telemedicine programs have the potential to increase health care quality, expand the specialties available to patients, and reduce the time and cost required to deliver care. We aimed to identify, validate, and prioritize unmet needs in the health care space of health worker-to-physician telemedicine programs and develop and refine a solution that addresses those needs. We collected information regarding user needs through ethnographic research, direct observation, and semistructured interviews with 37 stakeholders (n=5, 14% physicians; n=1, 3% public health program manager; n=12, 32% community health workers; and n=19, 51% patients) at 2 telemedicine clinics in rural West Bengal, India. We used the Spiral-Iterative Innovation Model to design and develop a prototype solution to meet these needs. We identified 74 unmet needs through our immersion in health worker-to-physician telemedicine programs. We identified a critical unmet need that achieving optimal teleconsultations in low- and middle-income countries often requires shifting tasks such as history taking and physical examination from high-skilled remote physicians to FHWs. To meet this need, we developed a prototype digital assistant that would allow FHWs to assume some of the tasks carried out by remote clinicians. The user needs of multiple stakeholder groups (patients, FHWs, physicians, and health organizations) were incorporated into the design and features of the task-shifting tool. The final prototype was shared with the health workers, physicians, and public health program managers who expressed that the tool would be useful and valuable. The final prototype that was developed was released as an open-source digital public good and may improve the quality and efficiency of care delivery in health worker-to-physician telemedicine programs.