Computational investigations of biological and soft-matter systems governed by strongly anisotropic interactions typically require resource-demanding methods such as atomistic simulations. However, these techniques frequently prove to be prohibitively expensive for accessing the long-time and large-length scales inherent to such systems. Conversely, coarse-grained models offer a computationally efficient alternative. Nonetheless, models of this type have seldom been developed to accurately represent anisotropic or directional interactions. In this work, we introduce a straightforward bottom-up, data-driven approach for constructing single-site coarse-grained potentials suitable for particles with arbitrary shapes and highly directional interactions. Our method for constructing these coarse-grained potentials relies on particle-centered descriptors of local structure that effectively encode dependencies on rotational degrees of freedom in the interactions. By using these descriptors as regressors in a linear model and employing a simple feature selection scheme, we construct single-site coarse-grained potentials for particles with anisotropic interactions, including surface-patterned particles and colloidal superballs in the presence of non-adsorbing polymers. We validate the efficacy of our models by accurately capturing the intricacies of the potential-energy surfaces from the underlying fine-grained models. Additionally, we demonstrate that this simple approach can accurately represent the contact function (shape) of non-spherical particles, which may be leveraged to construct continuous potentials suitable for large-scale simulations.