This paper proposes a low power consuming system for monitoring elderly people’s activities and their health conditions. The proposed system has two activity recognition modules: smartphone sensor-based wearable module; infrared grid sensor-based remote module. The two activity recognition modules work in a coordinated way. The fraction of the time the person is detected by the infrared sensor, the smartphone remains idle. As a result, energy consumption in the smartphone is reduced significantly, and hence the battery lifetime is increased. In the smartphone, a Feed-forward Neural Network (FNN) based activity recognition algorithm is implemented using fixed-point computation to further reduce energy consumption. A Convolutional Neural Network is used in the infrared sensor-based activity recognition module. The proposed system also has real-time health monitoring capability, which is based on ECG signal classification. A FNN leveraging fixed-point operation is used for ECG signal classification on an embedded ARM processor. Proposed fixed-point implementations of the FNNs are faster than floating-point implementation and require 50% less memory to store the neural network model parameters without loss of classification accuracy.