1 files changed, 108 insertions, 0 deletions

diff --git a/content/posts/2019-10-19-using-sentiment-analysis-for-clickbait-detection.md b/content/posts/2019-10-19-using-sentiment-analysis-for-clickbait-detection.md
new file mode 100644
index 0000000..8322e70
--- /dev/null
+++ b/content/posts/2019-10-19-using-sentiment-analysis-for-clickbait-detection.md
@@ -0,0 +1,108 @@
+---
+title: Using sentiment analysis for clickbait detection in RSS feeds
+url: using-sentiment-analysis-for-clickbait-detection-in-rss-feeds.html
+date: 2019-10-19T12:00:00+02:00
+type: post
+draft: false
+---
+
+## Initial thoughts
+
+One of the things that interested me for a while now is if major well
+established news sites use click bait titles to drive additional traffic to
+their sites and generate additional impressions.
+
+Goal is to see how article titles and actual content of article differ from each
+other and see if titles are clickbaited.
+
+## Preparing and cleaning data
+
+For this example I opted to just use RSS feed from a new website and decided to
+go with [The Guardian](https://www.theguardian.com) World news. While this gets
+us limited data (~40) articles and also description (actual content) is trimmed
+this really doesn't reflect the actual article contents.
+
+To get better content I could use web scraping and use RSS as link list and
+fetch contents directly from website, but for this simple example this will
+suffice.
+
+There are couple of requirements we need to install before we continue:
+
+- `pip3 install feedparser` (parses RSS feed from url)
+- `pip3 install vaderSentiment` (does sentiment polarity analysis)
+- `pip3 install matplotlib` (plots chart of results)
+
+So first we need to fetch RSS data and sanitize HTML content from description.
+
+```python
+import re
+import feedparser
+
+feed_url = "https://www.theguardian.com/world/rss"
+feed = feedparser.parse(feed_url)
+
+# sanitize html
+for item in feed.entries:
+    item.description = re.sub('<[^<]+?>', '', item.description)
+```
+
+## Perform sentiment analysis
+
+Since we now have cleaned up data in our `feed.entries` object we can start with
+performing sentiment analysis.
+
+There are many sentiment analysis libraries available that range from rule-based
+sentiment analysis up to machine learning supported analysis. To keep things
+simple I decided to use rule-based analysis library
+[vaderSentiment](https://github.com/cjhutto/vaderSentiment) from
+[C.J. Hutto](https://github.com/cjhutto). Really nice library and quite easy to
+use.
+
+```python
+from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
+analyser = SentimentIntensityAnalyzer()
+
+sentiment_results = []
+for item in feed.entries:
+    sentiment_title = analyser.polarity_scores(item.title)
+    sentiment_description = analyser.polarity_scores(item.description)
+    sentiment_results.append([sentiment_title['compound'], sentiment_description['compound']])
+```
+
+Now that we have this data in a shape that is compatible with matplotlib we can
+plot results to see the difference between title and description sentiment of an
+article.
+
+```python
+import matplotlib.pyplot as plt
+
+plt.rcParams['figure.figsize'] = (15, 3)
+plt.plot(sentiment_results, drawstyle='steps')
+plt.title('Sentiment analysis relationship between title and description (Guardian World News)')
+plt.legend(['title', 'description'])
+plt.show()
+```
+
+## Results and assets
+
+1. Because of the small sample size further conclusions are impossible to make.
+2. Rule-based approach may not be the best way of doing this. By using deep
+   learning we would be able to get better insights.
+3. **Next step would be to** periodically fetch RSS items and store them over a
+   longer period of time and then perform analysis again and use either machine
+   learning or deep learning on top of it.
+
+![Relationship between title and description](/assets/sentiment-analysis/guardian-sa-title-desc-relationship.png)
+
+Figure above displays difference between title and description sentiment for
+specific RSS feed item. 1 means positive and -1 means negative sentiment.
+
+[» Download Jupyter Notebook](/assets/sentiment-analysis/sentiment-analysis.ipynb)
+
+## Going further
+
+- [Twitter Sentiment Analysis by Bryan Schwierzke](https://github.com/bswiss/news_mood)
+- [AFINN-based sentiment analysis for Node.js by Andrew Sliwinski](https://github.com/thisandagain/sentiment)
+- [Sentiment Analysis with LSTMs in Tensorflow by Adit Deshpande](https://github.com/adeshpande3/LSTM-Sentiment-Analysis)
+- [Sentiment analysis on tweets using Naive Bayes, SVM, CNN, LSTM, etc. by Abdul Fatir](https://github.com/abdulfatir/twitter-sentiment-analysis)
+