This paper focuses on providing a solution to the direct conversion of speech to shorthand. Since shorthand is not understood by many but is used for writing quick transcripts, a product is developed that converts the speech to its appropriate Gregg shorthand. A website that will be used as a front end, will use a speech-to-text API to record the speech in real-time. The converted text will then be fed into a text-to-image retrieval model that derives its corresponding Gregg shorthand for the text. The text will then be displayed to the user in real-time. By achieving this, the model reduces the need to depend upon stenographers for transcribing scripts. The resulting model achieves a good result.