We propose a scalable in-band optical notch-filter labeling scheme for optical packet switching of high-bit-rate data packets. A detailed characterization of the notch-filter labeling scheme and its effect on the quality of the data packet is carried out in simulation and verified by experimental demonstrations. The scheme is able to generate more than 91 different labels that can be applied to 640-Gb/s optical time division multiplexed packets causing an eye opening penalty of $<$ 1.2-dB. Experimental demonstration shows that up to 256 packets can be uniquely labeled by employing up to eight notch filters with only 0.9-dB power penalty to achieve BER of 1E-9. Using the proposed labeling scheme, optical packet switching of 640 Gb/s data packets is experimentally demonstrated in which two data packets are labeled by making none and one spectral hole using a notch filter and are switched using a LiNbO $_3$ switch. The performance of the two data packets before/after switching is investigated by BER measurements and all 256 data channels achieve BER 1E-9 at the expense of only 4.1-dB total system penalty.