Owing to the increases of energy loads and penetration of renewable energy with variability, it is essential to determine the optimum capacity of the battery energy storage system (BESS) and demand response (DR) within the microgrid (MG). To accomplish the foregoing, this paper proposes an optimal MG operation approach with a hybrid method considering the game theory for a multi-agent system. The hybrid method operation includes both BESS and DR methods. The former is presented to reduce the sum of the MG operation and BESS costs using the game theory, resulting in the optimal capacity of BESS. Similarly, the DR method determines the optimal DR capacity based on the trade-off between the incentive value and capacity. To improve optimization operation, multi-agent guiding particle swarm optimization (MAG-PSO) is implemented by adjusting the best global position and position vector. The results demonstrate that the proposed approach not only affords the most economical decision among agents but also reduces the utilization cost by approximately 8.5%, compared with the base method. Furthermore, it has been revealed that the proposed MAG-PSO algorithm has superiority in terms of solution quality and computational time with respect to other algorithms. Therefore, the optimal hybrid method operation obtains a superior solution with the game theory strategy.