The principal objective of this paper is to estimate a nonlinear functional of state vector (NFS) in dynamical system. The NFS represents a multivariate functional of state variables which carries useful information of a target system for control. The paper focuses on estimation of the NFS in linear continuous-discrete systems. The optimal nonlinear estimator based on the minimum mean square error approach is derived. The estimator depends on the Kalman estimate of a state vector and its error covariance. Some challenging computational aspects of the optimal nonlinear estimator are solved by usage of the unscented transformation for implementation of the nonlinear estimator. The special quadratic functional of state vector (QFS) is studied in detail. We derive effective matrix formulas for the optimal quadratic estimator and mean square error. The quadratic estimator has a simple closed-form calculation procedure and it is easy to implement in practice. The obtained results we demonstrate on theoretical and practical examples with different types of an nonlinear functionals. Comparison analysis of the optimal and suboptimal estimators is presented. The subsequent application of the proposed optimal nonlinear and quadratic estimators demonstrates their effectiveness.