The texts in this article were partly composed with the help of artificial intelligence and corrected and revised by us. The following services were used for the generation:
Introduction
In today’s digital age, sentiment analysis is more important than ever. With the rise of social media and online reviews, customers have more opportunities than ever to share their thoughts and opinions about a company. This can be both a blessing and a curse - while positive reviews and feedback can help attract new customers, negative reviews can damage a company’s reputation and turn off potential customers.
Enter sentiment analysis. By analyzing customer feedback, social media posts, and other forms of textual data, companies can identify trends and patterns in customer sentiment. They can use this information to identify areas for improvement, respond to customer concerns, and even identify new opportunities for innovation and growth. But sentiment analysis is not just useful for businesses. It can be used in a wide range of industries and use cases, from political campaigns to healthcare and education. For example, political campaigns can use sentiment analysis to understand public opinion and tailor their messaging accordingly. Healthcare providers can use sentiment analysis to identify patient concerns and improve patient outcomes. Educators can use sentiment analysis to understand student opinions and improve educational outcomes.
In this post, we would like to take the opportunity to introduce a pre-trained transformer that has been trained for sentiment analysis - the Emotion Text Classifier.
General notes on the article
-
This article uses social media text messages to illustrate the capabilities of the model. Due to time constraints, the messages were not verified for accuracy and may contain inaccurate information.
-
In addition, the texts may be categorized differently depending on the recipient and their circumstances and experiences. In the following article, we have used our own interpretations as a starting point, which may not coincide with the interpretations of all readers.
Description of Training Data
This pre-trained transformer is a modification of the Emotion-English-DistilRoBERTa-Base model. 1 The task of this base model is to classify from text input what kind of emotion the user felt when writing the message. A number of datasets have been used for this purpose, such as the Emotion Dataset (Elvis et al.), GoEmotion (Demszky et al.), or MELD, (Poria et al.). 1 This provides users with a robust model for sentiment analysis. The difference between these two transformers is that the Emotion Text Classifier also uses an additional data set. This dataset is EmotionLines, which uses dialogues from the Friends series to augment the training data. 2
Model Usage
There are two ways to implement this model:
-
Pipeline: The pipeline offers us a particularly simple implementation that can be implemented quickly. However, information is lost as only the highest rated emotion is output.
-
Full Definition: The full definition is based on a definition we have already used in other articles, such as the one for T5. The definition of the model is more elaborate here, but we can look at the entire output instead of having already processed it. This way we can get the values for all the emotions in a message.
Below we show the application of the model for both cases as an example.
When analyzing the emotion of a message, context may be a necessary factor for proper classification. For example, if a message comes from a social media site and is to be processed, this message should be analyzed in the context of the dialogue and not as a single message. For the sake of simplicity, only single messages are considered in this article.
Pipeline
To use the pipeline, the first step is to import the pipeline class into our script.
from transformers import pipelineWe can then initialize our pipeline in “sentiment-analysis” mode for the “michellejieli/emotion_text_classifier” model. The actual transformer and tokenizer will be loaded automatically.
classifier = pipeline("sentiment-analysis", model="michellejieli/emotion_text_classifier")In the next step, we can use the pipeline directly to analyze a message for its associated emotions. All we have to do is call the instance with the message:
classifier("I love this!")Our output of this command now looks like this:
[{'label': 'joy', 'score': 0.9887555241584778}]Full Definition
For the complete definition of the model, we need a few more lines of code. We start here with the imports of transformers and torch.
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torchThen we have to download and initialize the tokenizer and transformer manually. This is done with the following code snippet:
tokenizer = AutoTokenizer.from_pretrained("michellejieli/emotion_text_classifier")
model = AutoModelForSequenceClassification.from_pretrained("michellejieli/emotion_text_classifier")For the sake of simplicity, we now define a method that makes the evaluation easier. This will also allow us to convert the evaluation into a single line later, as is the case with the pipeline. To convert our logits into a probability distribution, we need to use a softmax activation after processing.
def evaluate_input(input):
logits = tokenizer(input, return_tensors='pt')
outputs = model(**logits).logits
activation = torch.nn.Softmax(dim=1)
probs = activation(outputs)
print(probs)
print(model.config.id2label[int(probs.argmax())])Finally, we can analyze the same message as for the pipeline as an example.
evaluate_input("I love this!")Our results are as follows:
tensor([[1.7848e-03, 3.8670e-04, 4.7674e-04, 9.8876e-01, 2.7721e-03, 2.7513e-04,
5.5491e-03]], grad_fn=<SoftmaxBackward0>)
joyWe can now see that in addition to the most likely emotion, we can also analyze the probabilities for all other emotions. This is especially useful when there are two emotions in a message and no clear statement can be made.
In this case, we can simply display the result in a graph:
Results
In the following, we would like to present some examples that we have extracted from publicly available social networks. This allows us to get closer to the actual field of application - information from public sources - and thus provide comparable results. We have also taken care to include colloquial messages, such as those that use grammatically incorrect or incomplete sentences.
Due to the sheer number of messages posted daily on social networks, it was not always possible to find similarly complex messages for each emotion and thus achieve comparability of the model across emotions. However, we have included a number of examples for each emotion so that all readers can get an idea of the performance of the pre-trained transformer.
Anger
We are fucked aren’t we? Source
I will never call it X. Fuck you Elon. Source
Disgust
I’m so sick of this. I’ve been with my company for about a year now. Remote company which is a plus. Pay is fine. Benefits OK. But what I cannot stand are the customers. I constantly feel like an asshole because the company I work for is dropping the ball on stuff and since I’m the customer facing person I get shit on. Which yes, is sort of the job. But then some of them go complain to the board and it makes it look like I’m not doing shit. It’s super frustrating and I’m just tired of feeling dumb because I’m constantly trying to clean up the messes of other people. And I like being remote and the flexibility this job gives me. But idk I just feel like bad at my job all the time because my job depends on other people doing well. Im just sick of feeling this way. Source
It’s so painful. I’m working on this one app, it’s been developing over a year. It’s straight up an algorithm, not AI, but the new guys involved keep referring to it as AI. We keep explaining the difference and there isn’t one in their mind. “But it’s an AI powered engine isn’t it?” No you fucking rubes, it’s a series of calculations and simple if/then decisions made based on those calculations. Nowhere in there are we introducing AI. “But can it have a pirate voice?” Source
Yeah people don’t seem to realize professional programmers aren’t writing 200 line snippets. They are writing 200 line changes to a codebase of 200,000+ lines of code that it has to integrate with. I’m really tired of this joke that programmers just copy stuff of stack overflow all day and ChatGPT can do that now. The real world is not a data structures class. Source
This text shows quite well how many emotions can come together in a report. In this report, the pipeline would have produced very inaccurate results because a lot of information was not taken into account.
Joy
Today I found out that I passed my Calc II exam with an 88%! I’m happy that I got that high of a score! Source
A few years ago graduation was something I would never liked to hear about because I thought I failed and would never be able to be going to finish college, now I’m in my third semester and starting my fourth semester, and my GPA is good.. I have gone so far.., Source
I’m so proud of myself and happy to see some money in the bank I finally earned. Back in 2023 and most of 2022 I drank very excessively, didn’t do anything productive and hid in my room practically 24/7. Near the end it got to be such a mess there were 100s of dead maggots on the floor and even on the other half of my previous bed. Now I’m sober, in a clean room, working again, and should have an apartment all to myself very soon! For the first time in a long time I’m finally able to be proud of myself, and even excited for a future that isn’t pouring another cup of whiskey lol. Source
We found her about 10 years ago as a stray in NYC and she’s been with us ever since. She got out of a window once and was gone for over 24 hours but she came back, almost knocking at the front door. She’s been my little special buddy for so many years now and knowing I can still scoop her up and give her kisses warms my heart. She’s got the best personality and she’s very friendly and affectionate - basically the ideal cat. Snuggles with me like a teddy bear almost every night. She’s so perfect she has a heart marking on her side! I just needed to happyrant about my wonderful kitty. Gave her some love a moment ago and wanted to share it :) Source
Neutral
He’s doing what tech companies have always done: rise to prominence by exploiting legal grey areas, bootstrap your product using public data and open research, then loudly call for regulation while lobbying congress to help draft the new legislation. He’s trying to pull up the ladder and freeze out future competitors. The major record labels and tech companies like Google/YouTube did something similar with the DMCA. If you want to set up a service that hosts user-submitted content, you need to implement a hideously expensive infringement detection system to avoid being sued into oblivion. Small companies can’t afford to do that so power stays consolidated with the industry giants. And who lobbied for those requirements? You guessed it. Source
I don’t expect much philosophical or moral consistency. I think his position is that AI is potentially dangerous and needs controls and regulation, but also he’s not willing to pump the breaks because he wants money, and someone will do it anyway. He probably knows he’s a hypocrite, but like so many hypocrites today, he doesn’t care. Source
This is really it. Is AI going to be able to write better screenplays than a human? Not any time in the near future. That doesn’t mean the hedge fund bros running the industry aren’t going to try to use it to fuck humans out of decent pay to improve their margins. Source
Sadness
I have no communication skills. I have no friends, nobody. So alone in life. Source
I know that the future of 3D using AI isn’t quite here yet but the truth about that technology is that it’s development is exponential meaning that the rate of its improvement will speed up so it may seem far off but in reality it might be just around the corner. I’ve put so much hard work into learning 3D in the last few years but recently as I’m seeing more and more use of AI I’m starting to feel depressed and demotivated. I know some people will say it’s a way off or there’s nothing to worry about but I genuinely have stopped learning for the past month or so because I feel so discouraged. Does anyone else feel that way? Am I being needlessly pessimistic? Source
Surprise
Wow, this image looks so good! How did you do this?
This example was created by us and is therefore not referenced.
Wow, 10 comments in and I swear no one read the article. I was just listening to NPR talk about this hearing, so this was a good read to accompany it. He’s talking about fears of how AI will devolve society in ways worse than social media. Instead of creating the tech and then realizing the harm, he’s asking the harm be evaluated now and safeguards put in place. His specific example is the upcoming 2024 election. How AI will easily manipulate people, deep fake videos and sound bites that can be created with just a few minutes of input material. Bad actors at home and abroad can easily target and influence voters with hyper targeted content. Trust in society will breakdown. And what happens if a society loses all trust in its institutions? Source
Fear
I’m literally speechless WTF!! It all depends on how much time and how much it costs to render these videos, and how resource intensive it is… Source
I thought I was safe with my two dogs, but things would go left as time progressed. My mom and brother would only be gone for the weekend and as night time came every night, there would be a certain area in my house that I would get a negative sense of dread around my moms room. my sisters room was completely off limits while she was at work, and all I had was my own room and my mothers. Down the hall around just about in front of the bathroom I would suddenly catch goosebumps for no exact reason, and I thought nothing of it. The Saturday evening things started to worsen as I prepared for bed. I kept my dogs in my room with me every time I slept because of how scary I was and I even put the food bowls in my room until day time. I usually went to the bathroom before sleep so I didn’t have to step foot in that creepy hall in the middle of the night, and that same area as I mentioned before, I didn’t just get goosebumps, but for some reason that area was awfully hot. So I rushed in my room with the tablet that has access to our security systems. I knew something was up because something about that area didn’t make sense I mean, one spot of the house felt very hot but the rest of the house had a cool breeze flowing? I checked our security cameras and just as I expected nothing was there. Sunday night, I gained a powerful urge to check these cameras and so I did. After I checked everything, I put a knife under my pillow. I have the cameras a nice sweep once again. Then something triggered me to go out for whatever reason, I don’t remember. But I couldn’t help but notice my dog not taking her eye off the door. I called her about five times and she wouldn’t move a muscle. I checked out the peephole and saw a very black figure run by. It was so dark that not the cameras or the sensor light could catch it. I grew nauseous with fear and prepared for vomit as I felt it coming. I rushed to my room and tried to dial 911 but I couldn’t, my shivering was severe. Before I knew it, the thing left and I explained it the best I can to my mom but she quickly dismissed it and told me to stop playing with her. My brother reacted the same as expected. But I never stayed by myself ever again. Source
The text has been slightly shortened for simplicity, without removing important information.
Incorrect Classifications
Having presented a number of texts that could be well categorized into different areas of emotion, we will now look at a number of examples where the categorization was incorrect or inaccurate. We will start with the following text and its analysis:
How is a young designer supposed to build own brand and portfolio when their work is just burried under pile of ai generated images? Fine, use ai to speed up some parts of workflow but it still will never be as fast and as efficient as people spamming midjourney images burring your artwork. Source
This is an excerpt due to the length of the text.
In this text, “neutral” was identified as the predominant emotion. However, in our opinion, this text is much more infused with anger and disgust than neutral.
In addition to this example, we have prepared a second, clearer example.
When I was 5 yrs old, my family and I were living in an apartment which we didn’t find out until we moved that the complex was known to be haunted. The four of us shared a 1 bed and 1 bath. My sister and slept between our parents on one bed. This bed faced the door that we kept wide open. So, on this particular night, I had randomly woke up in the middle of the night. Something in me told to look at the door and there he was, standing still at the door, unmoving. This figure was a tall man in a vintage diving suit with the bowl shaped helmet that encased the head, but I realized he had no head… The circular windows in the helmet where the divers head could be seen, were completely see through. I could see the hallway behind him rather than a silhouette of a head. This freaked me out so much that I hid under my covers but daringly took another peek. He was still there in the exact same position. I looked at the door a third time to make sure I wasn’t crazy or something of my imagination. I was frozen with so much fear that I couldn’t even wake up my parents. I stayed beneath the covers until the sun came up. I couldn’t sleep a wink. That is one of my vivid ghost encounters that will forever haunt me. I have slept with my door completely shut ever since. Source
In our opinion, the emotion of “fear” is given far too little weight in this text, while “neutral” is listed as the dominant emotion.
Summary of results
The examples shown here allow an assessment of the quality of the results. In our opinion, the results shown here are realistic and representative, and the model has no problems with colloquial language or incomplete sentences.
We have also pointed out some errors in the model. It seems that the emotion “neutral” was rated too high in several texts. This slightly distorts the results in general, but we can also see misclassifications in some texts. This should be taken into account when using the pretrained transformer.
In our opinion, this model is a simple method to make representative statements about the emotions of text authors with little effort.
TL;DR
In this post we introduced the Emotion Text Classifier - an improvement of the popular Emotion-English-DistilRoBERTa-Base model. It allows us to identify the associated emotion of the author based on text messages. This makes it easier for individuals and companies, for example, to better tailor their services to feedback by categorizing feedback into different emotions at the time of posting. The model achieves qualitatively good results in the context of this task, although in our opinion the emotion “neutral” is sometimes weighted too heavily. Apart from this feature, this transformer seems to be a good basis for sentiment analysis.