Computer vision assisted diagnostic systems are gaining popularity in different healthcare applications. This paper presents a video analysis and pattern recognition framework for the automatic grading of vertical suspension tests on infants during the Hammersmith Infant Neurological Examination (HINE). The proposed vision-guided pipeline applies a color-based skin region segmentation procedure followed by the localization of body parts before feature extraction and classification. After constrained localization of lower body parts, a stick-diagram representation is used for extracting novel features that correspond to the motion dynamic characteristics of the infant's leg movements during HINE. This set of pose features generated from such a representation includes knee angles and distances between knees and hills. Finally, a time-series representation of the feature vector is used to train a Hidden Markov Model (HMM) for classifying the grades of the HINE tests into three predefined categories. Experiments are carried out by testing the proposed framework on a large number of vertical suspension test videos recorded at a Neuro-development clinic. The automatic grading results obtained from the proposed method matches the scores of experts at an accuracy of 74%.