Sign language plays an important part in the hearing/speech impaired person’s life as it is the prime medium of communication for them. However, not everyone understands the sign language hence, it becomes difficult for the disabled to communicate in day-to-day life. The sign language constitutes of various hand gestures which can represent various words and expressions. In this paper, the aim is to build a reliable communication interpretation program for interpreting Indian sign language and converting it to a readable output. The task is accomplished using Image processing and Machine learning. Our proposed project can find its applicability in the day to day for communication, it can also work for learning various gestures in gesture based automatic systems.