The automatic generation control (AGC) is one of the core control systems in power grids that regulate frequency within the permissible range. However, its dependence on communication makes it highly vulnerable to cyber-attacks. An arbitrary false data injection attack (FDIA) on AGC frequency and tie-line flow measurements will likely be detectable by bad data detection methods; however, if an attack can be launched optimally, it often becomes stealthy. In this regard, we develop a framework of optimal FDIAs (OFDIAs) to demonstrate the feasibility of such attacks in the power system frequency control loop. We propose a linearized formulation of discretized power systems’ dynamics in an optimization framework to model OFDIAs that compromise the AGC system by corrupting tie-line flow and generators’ frequency measurements. Using the proposed formal modeling, we study the effects of two types of FDIAs, continuous and time-limited, on the frequency behavior in power grids. The results demonstrate that continuous OFDIAs can lead to severe consequences on a power grid’s performance, such as frequency instability. In contrast, the time-limited FDIAs can cause the frequency to fluctuate beyond the acceptable range, which may lead to the triggering of the frequency-based protection relays.