Heart rate (HR) response to workout intensity reflects fitness and cardiorespiratory health. Physiological models have been developed to describe such heart rate dynamics and characterize cardiorespiratory fitness. However, these models have been limited to small studies in controlled lab environments and are challenging to apply to noisy—but ubiquitous—data from wearables. We propose a hybrid approach that combines a physiological model with flexible neural network components to learn a personalized, multidimensional representation of fitness. The physiological model describes the evolution of heart rate during exercise using ordinary differential equations (ODEs). ODE parameters are dynamically derived via a neural network connecting personalized representations to external environmental factors, from area topography to weather and instantaneous workout intensity. Our approach efficiently fits the hybrid model to a large set of 270,707 workouts collected from wearables of 7465 users from the Apple Heart and Movement Study. The resulting model produces fitness representations that accurately predict full HR response to exercise intensity in future workouts, with a per-workout median error of 6.1 BPM [4.4–8.8 IQR]. We further demonstrate that the learned representations correlate with traditional metrics of cardiorespiratory fitness, such as VO2 max (explained variance 0.81 ± 0.003). Lastly, we illustrate how our model is naturally interpretable and explicitly describes the effects of environmental factors such as temperature and humidity on heart rate, e.g., high temperatures can increase heart rate by 10%. Combining physiological ODEs with flexible neural networks can yield interpretable, robust, and expressive models for health applications.