With the increasing average life expectancy of world population, there are more and more dementia patients and the need of assistive technology emerges. According to the literature, the progress of cognitive impairment can be suppressed to be slower if the patients are constantly in calm mood. An effective way is suggested by keeping them in social relationship with the others. This paper presents a project with the goal of developing a framework for supporting dementia patients. This project features a conversational agent that can serve as a companion for dementia patients. Currently it has limited conversational ability but can generate back channel feedbacks, such as head nods and verbal acknowledgment, on the basis of acoustic information in the user's speech. In order to compensate the companionship of the agent and the ability to assist the user's memory, we are developing a memory vest which is equipped with portable devices including an Android smartphone, two IC audio recorders, and a digital video recorder to log the daily life of the patient. The gathered activity history database can then be used to enrich the dialogue ability of the agent and for helping the user to recall his / her own memory.