“If We Only Knew How You Feel”—A Comparative Study of Automated vs. Manual Classification of Opinions of Customers on Digital Media

Huoston Rodrigues Batista,José Carmino Gomes Junior,Renato José Sassi,Marcelo Drudi Miranda,Andréa Martiniano,Marcos Antonio Gaspar

doi:10.4236/sn.2019.81005

Abstract

The Web development has drastically changed the human interaction and communication, leading to an exponential growth of data generated by users in various digital media. This mass of data provides opportunities for understanding people’s opinions about products, services, processes, events, political movements, and organizational strategies. In this context, it becomes important for companies to be able to assess customer satisfaction about their products or services. One of the ways to evaluate customer sentiment is the use of Sentiment Analysis, also known as Opinion Mining. This research aims to compare the efficiency of an automatic classifier based on dictionary with the classification by human jurors in a set of comments made by customers in Portuguese language. The data consist of opinions of service users of one of the largest Brazilian online employment agencies. The performance evaluation of the classification models was done using Kappa index and a Confusion Matrix. As the main finding, it is noteworthy that the agreement between the classifier and the human jurors came to moderate, with better performance for the dictionary-based classifier. This result was considered satisfactory, considering that the Sentiment Analysis in Portuguese language is a complex task and demands more research and development.

Highlights

With the rise of Web 2.0 and the popularization of social networks and communication platforms, the number of people expressing opinions about products, services and their experiences tends to increase [1].This scenario represents an opportunity for companies to extract insights from this mass of unstructured data [2], and at the same time presents a challenge, considering that the mass of data being handled increases exponentially every day, making manual analysis impracticable [3].In addition, the marketing department of companies can benefit from these insights
This research aims to compare the efficiency of an automatic classifier based on dictionary with the classification by human jurors in a set of comments made by customers in Portuguese language
Automatic methods for dealing with the Sentiment Analysis usually involve lexical-based approaches [10] and machine learning [5], or a combination of both [6]. Both approaches have advantages and disadvantages, but none of them produces perfect results, which is perfectly acceptable, taking into account the limitations already discussed above. An example of this can be found in Canhoto and Padmanabhan [9] who undertook a comparative study of automated versus manual analysis of social media content

Summary

Introduction

With the rise of Web 2.0 and the popularization of social networks and communication platforms, the number of people expressing opinions about products, services and their experiences tends to increase [1]. This scenario represents an opportunity for companies to extract insights from this mass of unstructured data [2], and at the same time presents a challenge, considering that the mass of data being handled increases exponentially every day, making manual analysis impracticable [3]. Research shows that 81% of (American) users claim to have done online research on a product at least once. Since it is likely that there are thousands of these online product reviews, analysis of this content cannot be performed manually, and should be performed out automatically

Objectives

Methods

Findings

Conclusion