We introduce a new classification framework that combines the characteristics of matrix factorization with the discriminative capabilities of kernel methods. Short-time analysis of audio signals having different durations result in sets of feature vectors having different cardinalities. Support vector machines handle such varying-length feature sets using dynamic kernels, such as the intermediate matching kernel (IMK). IMK works by utilizing the so-called virtual vectors which select pairs of feature vectors to learn discrimination between classes. Existing formulations of IMK choose virtual vectors from the most information-bearing regions of classes, such as cluster means. This form of IMK completely ignores the feature vector pairs that lie around the class boundaries. To overcome this limitation, we propose an alternative formulation of IMK based on archetypal analysis (AA) and deep archetypal analysis (DAA). AA represents the data in terms of boundary elements, whereas DAA represents data in terms of both boundary and average elements. The proposed AA and DAA based intermediate matching kernel (AA/DAA-IMK) utilizes the elements generated from AA and DAA as the virtual feature vectors. Experimental evaluation on four different bioacoustic datasets show that the introduction of AA and DAA into the IMK framework leads to a noticeable improvement in classification accuracy.