This paper showcases a supervised machine learning classifier to bridge the gap between qualitative and quantitative research in media studies, leveraging recent advancements in data-driven approaches. Current machine learning methods make it possible to gain insights from large datasets that would be impractical to analyze with more traditional methods. Supervised document classification presents a good platform for combining specific domain knowledge and close reading with broader quantitative analysis. The study focuses on a dataset of 37 185 articles from the Finnish countermedia publication MVlehti, from which a randomly sampled 997 articles were annotated into three categories based on frame analysis. Contextual sequence representations from the finBERT language model, topic distributions from a trained topic model, and a structural, HTML-aware featureset developed in prior work are employed as classification features. The hypothesis that BERT-based embeddings could be improved upon by augmenting them with additional information is supported by recent promising results in natural language benchmarks and tasks (Peinelt, Nguyen, and Liakata 2020; Glazkova 2021). In our study, combining contextual embeddings with topics resulted in only marginal performance increases, and this improvement was observed mostly in minority classes. Despite this, potential future developments to achieve better classification performance are outlined. Based on the experiments, automated frame analysis with neural classifiers is possible, but the accuracy is not yet sufficient for inferences of high certainty.