Mainstream Internet of Things (IoT) techniques for smart homes focus on appliances and surveillance in smart cities. Most of the researchers utilize vision sensors in IoT environment targeting only adult users for various applications such as abnormal activity recognition. This paper introduces a new paradigm in vision sensor IoT technologies by analyzing the behavior of baby through an intelligent multimodal system. Traditional wearable sensors such as heartbeat if attached to any body part of the baby make him uncomfortable and also some babies are paranoid toward sensors. Our vision based baby monitoring framework employs one of the process improvement techniques known as control charts to analyze the baby behavior. We construct control chart in a specific interval for real-time frames generated by Raspberry Pi (RPi) with attached vision sensor. Baby motion is represented through points on control chart, if it exceeds upper control limit (UCL) or falls from lower control limits (LCL), it indicates abnormal behavior of the baby. Whenever such a behavior is encountered, a signal is transmitted to the interconnected devices in IoT as an alert to baby care takers in smart health care centers. Our proposed framework is adaptable, a single RPi can be used to monitor a baby in home or a network of RPi’s for an IoT in a children nursery for multiple babies monitoring. Performance evaluation on our own created dataset indicates the better accuracy and efficiency of our proposed framework.