Let A be a commutative k-algebra over a field of k and Ξ a linear operator defined on A. We define a family of A-valued invariants Ψ for finite rooted forests by a recurrent algorithm using the operator Ξ and show that the invariant Ψ distinguishes rooted forests if (and only if) it distinguishes rooted trees T, and if (and only if) it is finer than the quantity α( T)=|Aut( T)| of rooted trees T. We also consider the generating function U(q)=∑ n=1 ∞ U nq n with U n=∑ T∈ T n (1/α(T))Ψ(T) , where T n is the set of rooted trees with n vertices. We show that the generating function U( q) satisfies the equation Ξ exp U(q)=q −1U(q) . Consequently, we get a recurrent formula for U n ( n⩾1), namely, U 1= Ξ(1) and U n = ΞS n−1 ( U 1, U 2,…, U n−1 ) for any n⩾2, where S n(x 1,x 2,…) (n∈ N) are the elementary Schur polynomials. We also show that the (strict) order polynomials and two well-known quasi-symmetric function invariants of rooted forests are in the family of invariants Ψ and derive some consequences about these well-known invariants from our general results on Ψ. Finally, we generalize the invariant Ψ to labeled planar forests and discuss its certain relations with the Hopf algebra H P,R D in Foissy (Bull. Sci. Math. 126 (2002) 193) spanned by labeled planar forests.