A Quantitative Study of Software Reviews Using Content Analysis Methods

Abstract

Online product reviews play a critical role for consumers to make decision of product purchasing, and become an important data source for vendors to build a recommendation system. Some consumers even won’t buy a product without reading online reviews first, and some vendors invite reviewers to write product reviews before a product is released to the market. However, review quality could great impact the useless of the reviews for supporting the purchasing decision and recommendation. In order to produce enough high-quality reviews, it is a normal practice that vendors provide incentives for writing product reviews. Some research work showed the incentive reviews could be more bias comparing to organic reviews, but other work showed incentive reviews contain more useful information. However, some other research work showed very different results regarding the differences between the incentivized reviews and organic reviews. One of the reasons explaining the observation differences could be due to the quality of the reviewers and the reviews. Therefore, it is necessary to control the quality of the reviewers and the reviews to more objectively comparing the two types of reviews. In this research, we first discuss an approach for ensuring the quality of the data collection and processing to ensure the quality of the comparison study. Then we explore the differences between incentivized reviews and organic reviews collected from a website that provides reviews for enterprise software systems. Several parameters of the reviews such as overall score and sentiment and subjectivity of the reviews were analyzed and compared. The research result could provide a reference for appropriately using the reviews and managing the reviewing process.