Bitcoin is gaining ground in recent years. In the Bitcoin system, miners provide computing power to confirm transactions in pursuit of transaction fees, while users compete by bidding transaction fees for faster confirmation. This process is in essence analogous to online ad auctions, where advertisers bid for more prominent ad slots. Therefore, inspired by ad auction research, we propose to apply the Generalized Second Price (GSP) auction mechanism in the dynamic confirmation game on Bitcoin transactions. Our model is targeted to deal with the problems caused by instability and low efficiency in the currently-adopted Generalized First Price (GFP) auction model in Bitcoin confirmation games. Besides, we use the “rank-by-cost” rule to replace the “rank-by-fee” rule, where each transaction’s cost is calculated by the user-submitted fee and the waiting time. Aiming to probe users’ equilibrium strategy, we first discuss the GSP game with complete information under synchronous submissions, and show that it has the Locally Envy-Free equilibrium. Then, we study the GSP game with incomplete information under asynchronous submissions, and define two types of strategies, i.e., the Farsighted Balanced (FB) strategy and the Instant Balanced (IB) strategy. The FB strategy is in line with users’ practical needs of determining fees so as to maximize the long-term payoffs; however it cannot generate a stable equilibrium. Alternatively, the IB strategy focuses on the instant payoff maximization, and if all users follow the IB strategy, their equilibrium fees can finally converge to a stable profile. Finally, we design computational experiments to validate our theoretical models and analysis. Our research findings indicate that this novel GSP mechanism is superior to the currently adopted GFP mechanism. Besides, the convergence of the GSP game under the IB strategy has also been illustrated by the computational experiments.