The increasing demand for storing various types of goods has led to a raise in the need for storage capacity in warehousing systems. Autonomous vehicle storage and retrieval systems (AVS/RSs) offer high flexibility by allowing different configurations to meet different storage requirements. The system mainly completes operations through elevators and multiple rail-guided vehicles (RGVs). This paper focuses on the scheduling optimization of compound operations in the AVS/RS to improve system performance. Compound operations involve the coordinated execution of both single-command and double-command operations. A mathematical model with compound operations was proposed and effectively decomposed into a horizontal component for RGVs and a vertical counterpart for the elevator, which can represent the operations of one elevator cooperating with multiple RGVs. The goal of this model was to minimize the makespan for compound operations and to determine the optimal operation sequence and path for RGVs. An improved discrete particle swarm optimization (DPSO) algorithm called AGDPSO was proposed to solve the model. The algorithm combines DPSO and a genetic algorithm in an adaptive manner to prevent the algorithm from falling into local optima and relying solely on the initial solution. Through rigorous optimization, optimal parameters for the algorithm were identified. When assessing the performance of our improved algorithm against various counterparts, considering different task durations and racking configurations, our results showed that AGDPSO outperformed the alternatives, proving its effectiveness in enhancing system efficiency for the model. The findings of this study not only contribute to the optimization of AVS/RS but also offer valuable insights for designing more efficient warehouses. By streamlining scheduling, improving operations, and leveraging advanced optimization techniques, we can create a more robust and effective storage and retrieval system.