Elderly care is a significant livelihood project in the increasingly serious aging society. Nowadays, the wider application of the Internet of Things (IoT) technology on assistant means has making an importance contribution to elderly care in institutions and at home. On the broader data foundation collected by IoT devices, Human Activity Recognition (HAR) with its high demand in various elderly care applications also has grabbed considerable research attentions. However, the elderly’s sensitive data transmitted over the wireless communication channel need to address many security concerns, and the accuracy of activity recognition is susceptible to many influencing factors, especially, the employed feature selection method and classifier. In this paper, to ensure the confidentiality of the elderly’s sensitive data, a secure and efficient group-based key establishment and authentication framework is first proposed. Subsequently, activity recognition is investigated within the security data sensitive framework, where a feature reorganization based feature selection method is proposed with the demonstrated relationship between the recognition accuracy and strongly correlative features, and three classifiers are investigated based on classical ones. By contrast, an improved convolutional neural network (CNN) enabled classifier is singled out to conduct the activity recognition with the feature reorganization method. Finally, security proofs show that the proposed security framework can ensure data confidentiality and be resilient to possible attacks, and experimental analyses show investigations in activity recognition can achieve more cost-effective results.