Visual object tracking in unconstrained environments is a challenging task in computer vision. How to design an efficient discriminative feature representation is one challenging issue. To improve the adaptability of the tracker to large object appearance changes, the observation model needs to be updated online. However, a bad model update using inaccurate training samples can lead to model drift problem. Therefore, how to design an efficient online observation model and a model update strategy are two other challenging issues. This paper proposes the concatenation of histogram of oriented gradients variant (HOGv) and color histogram as the feature representation to balance discriminative power and efficiency. The single-hidden-layer feedforward neural network (SFNN) is used as an observation model, and the recursive orthogonal least squares (ROLS) algorithm is used to update the model online. A bidirectional tracking scheme is designed to alleviate the model drift problem during online tracking. The proposed bidirectional tracking scheme consists of three modules: the forward tracking module, the backward tracking module and the integration module. The forward tracking module first finds all the candidate regions, and then, the backward tracking module calculates the respective confidence of each candidate region according to historical information. Finally, the integration module integrates both of the first two modules' results to determine the final tracked object and the model update strategy for the current frame. Extensive evaluations of the existing tracking benchmarks have shown that the proposed tracking framework results in significant performance improvements compared with the base tracker, and it outperforms most of the state-of-the-art trackers.