We have developed the open-source toolbox custEM (customizable electromagnetic modeling) for the simulation of complex 3D controlled-source electromagnetic (CSEM) problems. It is based on the open-source finite-element library FEniCS, which supports tetrahedral meshes, multiprocessing, higher order polynomials, and anisotropy. We use multiple finite-element approaches to solve the time-harmonic Maxwell equations, which are based on total or secondary electric field and gauged potential formulations. In addition, we develop a secondary magnetic field formulation, showing superior performance if only magnetic fields are required. Using Nédélec basis functions, we robustly incorporate the current density on the edges of the mesh for the total field formulations. The latter enable modeling of CSEM problems taking topography into account. We evaluate semianalytical 1D layered-earth solutions with the pyhed library, supporting arbitrary configurations of dipole or loop sources for secondary field calculations. All system matrices have been modified to be symmetric and solved in parallel with the direct solver MUMPS. Aside from the finite-element kernel, mesh generation, interpolation, and visualization modules have been implemented to simplify and automate the modeling workflow. We prove the capability of custEM, including validation against analytic-solutions, crossvalidation of all implemented approaches, and results for a model with 3D topography with four examples. The object-oriented implementation allows for customizable modifications and additions or to use only submodules designed for special tasks, such as mesh generation or matrix assembly. Therefore, the toolbox is suitable for crossvalidation with other codes and as the basis for developing 3D inversion routines.