Abstract Ultrafiltration membrane is a convenient system to fractionate fish protein hydrolysate and recovery of peptide fraction with high ACE inhibitory activity. However, the limitation of applying membrane system in protein fractionation is its poor selectivity. This study aiming to assess the application of multilayer ultrafiltration in order to improve the selectivity of the tilapia by-product protein hydrolysate (TBH) separation by achieving higher amount of small peptides. Flat sheet regenerated cellulose (RC) membrane with molecular weight cut-off (MWCO) of 10 and 5 kDa were used for the separation of peptide mixtures. The membrane were arranged in the orientation of 10/5 kDa and 5/5 kDa, in which these two membrane were stacked together in one device. The performance of multilayer membrane were evaluated based on the permeate flux and peptide transmission, and compare with the single membrane system (5 and 10 kDa). The performance ultrafiltration membrane for fractionating TBH was studied under two different conditions (rotation speed and pH). The highest permeate flux and peptide transmission were obtained at membrane with pore size of 10 kDa (single), followed by 10/5 (multilayer), 5 (single) and 5/5 kDa (multilayer). Based on selectivity analysis, most permeate produced were composed of peptides lower than 1500 Da. When the smaller membrane’s pore size is used (5 kDa membrane and 5/5 multilayer membranes), the amount of small-sized peptide (