This paper focuses on the optimal minimum mean square error estimation of a nonlinear function of state (NFS) in linear Gaussian continuous-time stochastic systems. The NFS represents a multivariate function of state variables which carries useful information of a target system for control. The main idea of the proposed optimal estimation algorithm includes two stages: the optimal Kalman estimate of a state vector computed at the first stage is nonlinearly transformed at the second stage based on the NFS and the minimum mean square error (MMSE) criterion. Some challenging theoretical aspects of analytic calculation of the optimal MMSE estimate are solved by usage of the multivariate Gaussian integrals for the special NFS such as the Euclidean norm, maximum and absolute value. The polynomial functions are studied in detail. In this case the polynomial MMSE estimator has a simple closed form and it is easy to implement in practice. We derive effective matrix formulas for the true mean square error of the optimal and suboptimal quadratic estimators. The obtained results we demonstrate on theoretical and practical examples with different types of NFS. Comparison analysis of the optimal and suboptimal nonlinear estimators is presented. The subsequent application of the proposed estimators demonstrates their effectiveness.