Crosstalk between histone modifications represents a fundamental epigenetic mechanism in gene regulation. During the transcription elongation process, the histone deacetylase complex Rpd3S is recruited to H3K36-methylated nucleosomes to suppress cryptic transcription initiation. However, how subunits of Rpd3S are assembled and coordinated to recognize nucleosomal substrates and exert their deacetylation function remains unclear. Here we report the structure of Saccharomyces cerevisiae Rpd3S deacetylase bound to H3K36me3-modified nucleosome at 3.1 Å resolution. It shows that Sin3 and Rco1 subunits orchestrate the assembly of the complex and mediate its contact with nucleosome at multiple sites, with the Sin3-DNA interface as a pivotal anchor. The PHD1 domain of Rco1 recognizes the unmodified H3K4 and places the following H3 tail toward the active site of Rpd3, while the chromodomain of Eaf3 subunit recognizes the H3K36me3 mark and contacts both nucleosomal and linker DNA. The second copy of Eaf3-Rco1 is involved in neighboring nucleosome binding. Our work unravels the structural basis of chromatin targeting and deacetylation by the Rpd3S complex.