by Maria Korolov

What is sentiment analysis? Using NLP and ML to extract meaning

Feature
Sep 09, 2021
AnalyticsMachine LearningNatural Language Processing

Sentiment analysis, which enables companies to determine the emotional value of communications, is now going beyond text analysis to include audio and video.

artificial intelligence automation digital brain thinkstock 875595818 100749926 orig
Credit: IBM

Sentiment analysis defined

Sentiment analysis is analytical technique that uses statistics, natural language processing, and machine learning to determine the emotional meaning of communications.

Companies use sentiment analysis to evaluate customer messages, call center interactions, online reviews, social media posts, and other content. Sentiment analysis can track changes in attitudes towards companies, products, or services, or individual features of those products or services.

Sentiment analysis examples

One of the most prominent examples of sentiment analysis on the Web today is the Hedonometer, a project of the University of Vermont’s Computational Story Lab.

The group analyzes more than 50 million English-language tweets every single day, about a tenth of Twitter’s total traffic, to calculate a daily happiness store.

The approach is computationally straightforward. The lab collected a set of 10,000 commonly used words and, via Amazon’s Mechanical Turk service, had people give each word a happiness score of one to nine. Neutral words, and words that depend heavily on context, are filtered out, and the scores of the rest are added up and averaged to determine daily happiness scores. The word lists, with scores, are available on the project’s website in English and nine other languages.

This “bag of words” approach is an old-school way to perform sentiment analysis, says Hayley Sutherland, senior research analyst for conversational AI and intelligent knowledge discovery at IDC. “But it can be great for really large sets of text,” she says.

The Hedonometer also uses a simple positive-negative scale, which is the most common type of sentiment analysis.

While the Hedonometer uses a scale of one to nine, other approaches use three values — positive, negative, and neutral — or give a percentage store. More fine-grained approaches can also detect other emotions, Sutherland says. “It varies by tool,” she says. “Sad, angry, and excited are some of the more common ones.”

Companies can use this more nuanced version of sentiment analysis to detect whether people are getting frustrated or feeling uncomfortable.

Another type of sentiment analysis is intent detection. “You’re understanding what action they’re going to take,” she says. “For example, in sales, are they interested or are they not interested in buying?”

Sentiment analysis can do more than just look at plain text. “Some use facial analysis, some use vocal signals,” Sutherland says. “Increasingly, I’m seeing companies that are really focused on emotional AI. By understanding tone of voice in addition to what people actually say, it’s easier to understand sarcasm, for example.”

Public examples of sentiment analysis abound. The Obama administration used sentiment analysis to measure public opinion. The World Health Organization’s Vaccine Confidence Project uses sentiment analysis as part of its research, looking at social media, news, blogs, Wikipedia, and other online platforms.

This spring Google Cloud released its Intelligent Impact Solution, which includes a sentiment analysis component so that government agencies can better target their communication efforts and understand changing beliefs and behaviors surrounding COVID-19 vaccinations.

“It helps different states and municipalities to inform their COVID vaccination strategies,” says Sutherland.

Sentiment analysis tools

The basic level of sentiment analysis involves either statistics or machine learning based on supervised or semi-supervised learning algorithms. As with the Hedonometer, supervised learning involves humans to score a data set. With semi-supervised learning, there’s a combination of automated learning and periodic checks to make sure the algorithm is getting things right.

Deep learning is another means by which sentiment analysis is performed. “Deep learning uses many-layered neural networks that are inspired by how the human brain works,” says IDC’s Sutherland. This more sophisticated level of sentiment analysis can look at entire sentences, even full conversations, to determine emotion, and can also be used to analyze voice and video.

All the big cloud players offer sentiment analysis tools, as do the major customer support platforms and marketing vendors. Conversational AI vendors also include sentiment analysis features, Sutherland says.

Companies interested in performing sentiment analysis should first look at the tools and technologies they are already using, says Boris Evelson, vice president and principal analyst at Forrester Research. “Do they have a survey tool that has sentiment analysis in it? Customer feedback management platforms also have either basic or pretty decent sentiment analysis.”

There are also general-purpose analytics tools, he says, that have sentiment analysis, such as IBM Watson Discovery and Micro Focus IDOL.

“We advise our clients to look there next since they typically need sentiment analysis as part of document ingestion and mining or the customer experience process,” Evelson says.

Sentiment analysis, NLP, and ML

Few companies build their own sentiment analysis platforms. It requires in-house expertise and large training data sets. But it can pay off for companies that have very specific requirements that aren’t met by existing platforms. In those cases, companies typically brew their own tools starting with open source libraries.

NLP libraries capable of performing sentiment analysis include HuggingFace, SpaCy, Flair, and AllenNLP. In addition, some low-code machine language tools also support sentiment analysis, including PyCaret and Fast.AI.

For deep learning, sentiment analysis can be done with transformer models such as BERT, XLNet, and GPT3. GPT3 can even perform sentiment analysis with no training data.

Building their own platforms can give companies an edge over the competition, says Dan Simion, vice president of AI and analytics at Capgemini.

“This is the trend we are seeing in the marketplace in the big companies,” he says. “Otherwise, if you are buying the same tool off the shelf as your competitor, you don’t get the competitive advantage.”

Sentiment analysis APIs

A more common approach for companies building their own platforms is to pull in sentiment analysis functionality via APIs. All the major cloud providers offer this service — Amazon Comprehend, Azure Cognitive Services, and Google Natural Language API to name a few. IBM Watson also has an API.

“The question is, at the end of the day, is how good these APIs are,” says Capgemini’s Simion. “If you have a niche product, they will have a hard time being useful.”

And then there’s the cost, he adds.

“Each and every API call costs money,” he says. “You have to make sure that it’s economically feasible to call on these particular APIs. But it could be a good solution for small and medium businesses.”

Sentiment analysis datasets

Machine language and deep learning approaches to sentiment analysis require large training data sets. Commercial and publicly available tools often have big databases, but tend to be very generic, not specific to narrow industry domains.

“You really need billions of words and examples to learn how to properly suss out the sentiment,” says Bryan Richardson, associate partner at McKinsey & Co.

The very largest companies may be able to collect their own given enough time.

“For a single company, retail or bank, it’s hard for them to get enough data on their own customers to build a model,” he says. “But now through tools like transfer learning, NLP models are being calibrated against very wide corpuses with billions and billions of records and then applied to different use cases.”

That means that a company with a small set of domain-specific training data can start out with a commercial tool and adapt it for its own needs.

Top sentiment analysis use cases

The biggest use case of sentiment analysis in industry today is in call centers, analyzing customer communications and call transcripts.

For example, if negative sentiment increases after a new product release, that could be an early indication that something is going wrong, enabling the company to do a deep dive to understand which features are causing problems or to get more agents on board to handle problems.

With customer support now including more web-based video calls, there is also an increasing amount of video training data starting to appear.

The same kinds of technology used to perform sentiment analysis for customer experience can also be applied to employee experience. For example, consulting giant Genpact uses sentiment analysis with its 100,000 employees, says Amaresh Tripathy, the company’s global leader of analytics.

“We use an AI tool, a conversational chatbot,” he says. “Instead of HR people reaching out to everyone to check if everything is going well and they’re feeling supported, there’s a chatbot, and you can choose whether to talk to the chatbot or not.”

This can help a company find areas where employees are struggling or not feeling supported. “We are looking for areas where we can potentially help,” he says. “We go in and have a conversation. It’s massively beneficial because we know how to provide support to people in a beneficial manner.”

Sentiment analysis can also be used for brand management, to help a company understand how segments of its customer base feel about its products, and to help it better target marketing messages directed at those customers.

“It is particularly useful in public relations,” says Andy Thurai, vice president and principal analyst at Constellation Research. “You want to know sooner rather than later if someone says something negative so you can do crisis management. Or, if you run a commercial, you can know if people like it or not.”

The old approach was to send out surveys, he says, and it would take days, or weeks, to collect and analyze the data.

“But people seem to give their unfiltered opinion on Twitter and other places,” he says.

More on sentiment analysis: