This study presents a unified formulation of topology optimization with a finite strain nonlocal damage model using the continuous adjoint method. For the primal problem to describe the material response including deterioration, we consider the standard Neo–Hookean constitutive model and incorporate crack phase-field theory for brittle fracture within the finite strain framework. For the optimization problem, the objective function is set to accommodate multiple objectives by weighting each sub-function, and the continuous adjoint method is employed to derive the sensitivity. Thus, both the governing equations for primal and adjoint problems are written as strong forms and hold at any moment and at any location in the continuum body or on its boundary. Accordingly, the proposed formulation is independent of the requirements from numerical implementation, such as element type or discretization method. In addition, the reaction–diffusion equation is used to update the design variable in an optimizing process, by which the continuous distribution of the design variable, as well as material properties, are realized. After the basic performance of the proposed formulation is demonstrated with a simple numerical setup, two-material (matrix and inclusion materials) and single-material (material and null) topology optimizations are presented, and discussions are made.