Energy harvesting promises to power billions of Internet-of-Things devices without being restricted by battery life. Since the energy harvester generally outputs weak and unstable energy, the system may suffer frequent and unpredictable power failures, thus falling into cyclically reboots without forward progress. Task-based intermittent computing system which periodically backs up system states into Non-Volatile Memory (NVM) is proposed to solve the non-progress problem, with the non-trivial cost of frequent backups. How to reduce the backup overhead becomes a major research problem for intermittent computing. Our paper, for the first time, proposes to parallelize state backup and program execution with asynchronous Direct Memory Access (DMA) to hide the backup latency into the program’s execution. But, straightforwardly executing the state backup and the program in parallel may cause inconsistent system state. In specific, the system state may be modified by the program during backup, and therefore may be backed up incorrectly and further cause the system to deliver an incorrect computation result. We make deep analysis on the system behavior and observe that, although the system state may be backed up incorrectly, the incorrect backup will be covered by the subsequent correct backups soon as the backup operations are performed frequently. In addition, only a small part of variables among all the program states may cause incorrect computation result. So, in this paper, we aggressively allow incorrect backups to occur, and propose a backup error detection method and a fault-tolerant backup management to guarantee the correctness of the system’s execution. To augment the parallel backup method, an adaptive execution method is further proposed to reduce the number of backups and balance the ratio between task execution time and backup latency. We design a run-time system to implement the proposed approach, and experimental results conducted on an STM32F7-based platform show that the proposed method can achieve a 2.6× average speedup.