The precise monitoring of fish keypoints and behavioral patterns is crucial in fish farming, influencing decisions on feeding schedules and assessing fish health. Traditional approaches to multi-object pose estimation often lean towards either Bottom-up or Top-down methods. Our innovative solution introduces a streamlined single-stage multi-object pose estimation framework, utilizing a unique approach to instance assignment based on location. By incorporating position encoding as an index for candidate pose heatmap groups, we achieve end-to-end multi-object pose estimation with reduced redundancy through non-maximum suppression. Our framework, named POLO, has been validated on a meticulously annotated fish keypoint dataset, demonstrating outstanding performance with a remarkable 65.34% OKS AP at 71.4 FPS on Tesla v100. With its real-time capabilities, POLO is highly adaptable, making it suitable for deployment on various edge computing devices and addressing real-world challenges effectively. We believe our framework can serve as a solid baseline for diverse pose estimation tasks across different domains.