We prove that topological recursion applied to the spectral curve of colored HOMFLY-PT polynomials of torus knots reproduces the n-point functions of a particular partition function called the extended Ooguri-Vafa partition function. This generalizes and refines the results of Brini-Eynard-Marino and Borot-Eynard-Orantin. We also discuss how the statement of spectral curve topological recursion in this case fits into the program of Alexandrov-Chapuy-Eynard-Harnad of establishing the topological recursion for general weighted double Hurwitz numbers partition functions (a.k.a. KP tau-functions of hypergeometric type).