Abstract This essay, derived from a lecture at “The Physics Modeling of Thought” workshop in Berlin in winter 2023, explores the mutually beneficial relationship between theoretical neuroscience and statistical physics through the lens of efficient coding and computation in cortical circuits. It highlights how the study of neural networks has enhanced our understanding of complex, nonequilibrium, and disordered systems, while also demonstrating how neuroscientific challenges have spurred novel developments in physics. The paper traces the evolution of ideas from seminal work on chaos in random neural networks to recent developments in efficient coding and the partial suppression of chaotic fluctuations. It emphasizes how concepts from statistical physics, such as phase transitions and critical phenomena, have been instrumental in elucidating the computational capabilities of neuronal networks in the brain. By examining the interplay between order and disorder in neural computation, the essay illustrates the deep connection between theoretical neuroscience and the statistical physics of nonequilibrium systems. This synthesis underscores the ongoing importance of interdisciplinary approaches in advancing both fields, offering fresh perspectives on the fundamental principles governing information processing in the brain. This multidisciplinary approach not only advances our understanding of neural computation and complex systems but also points toward future challenges at the intersection of neuroscience and physics.