Curated Datasets for Use in Automated Media Monitoring and Feedback System: “News Classification System” Dataset, “Government News Classification” Dataset

Abstract

Online journalism in India, a growing field that involves news websites and Digital media, connects with the Press Information Bureau (PIB), a government agency dedicated to sharing accurate information about government policies and initiatives with journalists. While various news outlets publish diverse articles and opinions on these topics, the government seeks to leverage Artificial Intelligence and Machine Learning for gathering feedback in multiple languages. To develop such a system, a notable obstacle is the lack of a readily accessible standard dataset is required. To address this, two datasets are developed named, 'NCS' and 'GNC,' consisting of information from 2020 to 2023 and collected through web scraping tools like Parsehub and manually scrapping. NCS represents News Classification system dataset and GNC represents Government News Classification. The 'NCS' dataset includes Indian news in Hindi, Marathi, and English with categorization of Indian news as government-related or not. Then, a Machine Learning model called "Government News Classifier" to sort news articles using the 'NCS' dataset into either government-related or non-government-related categories. The objective is to use this model to figure out if a news source is discussing topics related to the government or not. Using this model, we created the 'GNC' dataset, which contains only news articles related to government schemes and policies in Hindi, Marathi, and English. In GNC dataset, Human experts manually classify each news source into three categories: "government favourable," "government non-favourable," or "neutral." In essence, this research emphasizes the importance of having access to a large dataset, which can stimulate more advanced prediction models in this complex field.