Along with the environmental protection policies becoming strict in China, the air pollution control devices (especially selective catalytic reduction (SCR)) are widely equipped in coal-fired power plants. The installation and run of these devices will inevitably affect mercury (Hg) species distribution in coal fired by-products such like fly ash (FA) and gypsum. In this work, a new on-line coupling system based on atomic fluorescence spectrometry (AFS) with a home-made chromatographic workstation was successfully developed to identify Hg species through thermal programmed desorption (TPD). The influences of matrix, furnace temperature, and carrier gas flow on analytical performance were investigated and the parameters were optimized. The FA and gypsum samples from coal-fired power plants equipped with SCR were collected and the mercury species were analyzed by the developed coupling system. HgCl2 and HgO were the main species in FA, while Hg2Cl2 and HgO were the main species in gypsum. All of Hg species in the studied FA and gypsum samples were released below 400 °C. A sequential extraction procedure was applied to further verify the operational Hg species including mobile and non-mobile fractions in FA and gypsum samples. This study demonstrated that AFS coupled with TPD procedure was an effective method to analyze Hg species in coal combustion by-products from power plants.