Direct thermal-to-electric energy converters typically operate in the linear regime, where the ratio of actual maximum power relative to the ideal maximum power, the so-called fill factor (FF), is 0.25. Here, we show, based on fundamental symmetry considerations, that the leading order nonlinear terms that can increase the FF require devices with broken spatial symmetry. Studying nonlinear, thermoelectric transport across an asymmetric energy barrier defined in a single semiconductor nanowire, we find in both experiment and theory that we can increase the FF as well as maximum power. Geometric symmetry breaking combined with the design of nonlinear behavior thus represents a strategy for increasing the performance of thermoelectric or hot-carrier devices.