Organic matter is a critical factor which regulates nitrogen loss pathways of denitrification and anammox for microbes in marine ecosystems. However, only a little attention has been paid to contrasting studies on denitrification and anammox in sandy and muddy sediments, especially in the coastal continental shelf dominated by sandy sediments. This study determined the bulk properties and associated microbial nitrogen transformation processes of surface sediments in the East China Sea coastal shelf, with the aim of gaining insight into the interaction of nitrogen loss with organic matter at the molecular level. The results illustrate that nitrogen loss dominates in organic-rich muddy sediments, and its denitrification rate (14.39 nmol N g−1 h−1) and anammox rate (2.73 nmol N g−1 h−1) are greater than those of sandy sediments (denitrification rate = 5.55 nmol N g−1 h−1, anammox rate = 1.57 nmol N g−1 h−1). Furthermore, determination of the mean summed ladderanes shows higher anammox generated in the muddy sediments with a value of 167.78 ng g−1dw. Quantitative analysis demonstrated that organic-rich muddy sediments enhanced the copy number of the denitrifying functional gene nosZ and anammox functional gene hzsB. We inferred that the greater rate of nitrogen loss in muddy sediments was due to the coupling relationship between anammox and denitrification. Overall, the community distribution and abundance of denitrifying bacteria and anammox bacteria changed intricately under the influence of organic matter. Moreover, this study further improves the understanding of nitrogen loss pathways and mechanistic factors from sediments.