Objective. In this paper, a novel methodology for feature extraction to enhance classification accuracy of functional near-infrared spectroscopy (fNIRS)-based two-class and three-class brain–computer interface (BCI) is presented. Approach. Novel features are extracted using vector-based phase analysis method. Changes in oxygenated and de-oxygenated ) haemoglobin are used to calculate four novel features: change in cerebral blood volume (), change in cerebral oxygen exchange (), vector magnitude (|L|) and angle (k). is the sum and is difference of and , whereas |L| is magnitude and k is angle of vector. fNIRS signals of seven healthy subjects, corresponding to left-hand index finger tapping (LFT), right-hand index finger tapping (RFT) and rest are acquired from motor cortex using multi-channel continuous-wave imaging system. After removing physiological and instrumental noises from the acquired signals, the four novel features are calculated. For validation, conventional temporal, spatial and spatiotemporal features; mean, peak, slope, variance, kurtosis and skewness are also calculated using and. All possible two-feature and three-feature combinations of the novel and conventional features are then used to classify two-class (LFT vs RFT) and three-class (LFT vs RFT vs rest) fNIRS-BCI using linear discriminant analysis. Main results. Results demonstrate that combination of four novel features yields significantly higher average classification accuracies of 98.7 ± 1.0% and 85.4 ± 1.4% as compared to 68.7 ± 6.9% and 53.6 ± 10.6% using conventional features for two-class and three-class problem, respectively. Validation of proposed method on an open access database containing RFT, LFT and dominant side foot tapping tasks for 30 subjects also shows improvement in average classification accuracies for two-class and three-class fNIRS-BCIs. Significance. This study provides a step forward in improving the classification accuracies of state-of-the-art fNIRS-BCIs by showing significant improvement in classification accuracies of two-class and three-class fNIRS-BCIs using novel features extracted by vector-based phase analysis.