Batch Job Active-Active Resiliency System and Method

Abstract

This manuscript presents a robust solution for orchestrating batch job workflows in an active-active resiliency mode across multiple regions. The proposed system abstracts the complexities of job and step/task state management, provides automated and customizable retries, and ensures auditability and observability. The system maintains operational continuity and fault tolerance, leveraging lock marker files and a configurable scheduler, even during regional failures. This approach addresses the challenges of asynchronous workflow management, offering enhanced visibility, control, and efficiency in processing batch jobs.