Accurate segmentation of the left atrium (LA) from late gadolinium-enhanced cardiac magnetic resonance (LGE CMR) images is crucial for aiding the treatment of patients with atrial fibrillation. Few-shot learning holds significant potential for achieving accurate LA segmentation with low demand on high-cost labeled LGE CMR data and fast generalization across different centers. However, accurate LA segmentation with few-shot learning is a challenging task due to the low-intensity contrast between the LA and other neighboring organs in LGE CMR images. To address this issue, we propose an Adaptive Dynamic Inference Network (ADINet) that explicitly models the differences between the foreground and background. Specifically, ADINet leverages dynamic collaborative inference (DCI) and dynamic reverse inference (DRI) to adaptively allocate semantic-aware and spatial-specific convolution weights and indication information. These allocations are conditioned on the support foreground and background knowledge, utilizing pixel-wise correlations, for different spatial positions of query images. The convolution weights adapt to different visual patterns based on spatial positions, enabling effective encoding of differences between foreground and background regions. Meanwhile, the indication information adapts to the background visual pattern to reversely decode foreground LA regions, leveraging their spatial complementarity. To promote the learning of ADINet, we propose hierarchical supervision, which enforces spatial consistency and differences between the background and foreground regions through pixel-wise semantic supervision and pixel-pixel correlation supervision. We demonstrated the performance of ADINet on three LGE CMR datasets from different centers. Compared to state-of-the-art methods with ten available samples, ADINet yielded better segmentation performance in terms of four metrics.