Surface-enhanced Raman scattering (SERS) has proven to be a powerful diagnostic technique for conservation science. In this work, a novel swab-based detection strategy is proposed for minimally invasive SERS analysis of synthetic organic dyes in the cultural heritage field, employing flexible paper-based SERS probes prepared with a facile, cost-efficient and reproducible recipe. It was demonstrated that Ag nanostructures/paper probes composed of interconnected Ag nanostructures immobilized on unmodified cellulose paper showed excellent SERS enhancement for the investigated synthetic organic dyes. The high flexibility and conformability of paper-based substrate allows to extract minimal amounts of dyes from the work of art by gently swabbing in a dry-state condition, thereby avoiding potential damage to colorants from close contact with solvents such as water. Dyes could be transferred via swab-based extraction method without removing any visible fragments of the paint layer, which facilitates preserving the physical integrity of the artwork. SERS detection was subsequently performed directly on flexible substrate surfaces after micro-extraction of dyes. The minimally invasive nature of this strategy has further been confirmed by the spectrophotometric color measurements and successfully applied to the identification of synthetic organic dyes on real paper artworks---Chinese woodblock New Year paintings from the late Qing Dynasty. These results highlight the great potential of flexible paper-based Ag substrates based on swab-detection method for minimally invasive analysis of organic dyes in cultural heritage.