Twitter hashtag analysis: #edsclimatechat OR #ccbc18 OR #ccbc2018
Ben Anderson (b.anderson@soton.ac.uk, ben.anderson@otago.ac.nz, @dataknut)
Last run at: 2018-10-12 17:40:01 (Pacific/Auckland)
1 TL;DR
Just a bit of dataknut fun woven around the day job.
You’ll be wanting Section 6 for the trending hashtags…
2 Terms of re-use
2.1 License
CC-BY unless otherwise noted.
2.2 Citation
- Anderson, B (2018) Twitter hashtag analysis: #edsclimatechat OR #ccbc18 OR #ccbc2018. Downloaded from: https://dataknut.github.io/hashTagR//Users/ben/github/dataknut/hashTagR/docs/ccbc2018.html.
2.3 Code
See:
- Code: https://github.com/dataknut/hashTagR/blob/master/analysis/genericHashTagReport.Rmd
- Code/report history
- Raise an issue here
3 Purpose
The idea is to extract and visualise tweets and re-tweets of #edsclimatechat OR #ccbc18 OR #ccbc2018 (see www.climateandbusiness.com).
Why? Err…. Just. Because.
4 How it works
Code borrows extensively from https://github.com/mkearney/rtweet
The analysis used rtweet to ask the Twitter search API to extract ‘all’ tweets containing the #edsclimatechat OR #ccbc18 OR #ccbc2018 hashtags in the ‘recent’ twitterVerse.
It is therefore possible that not quite all tweets have been extracted although it seems likely that we have captured most recent human tweeting which was the main intention. Future work should instead use the Twitter streaming API.
## [1] "Found 8 files matching #edsclimatechat OR #ccbc18 OR #ccbc2018 in ~/Data/twitter/"The data has:
- 964 tweets (including 24 quotes and 524 re-tweets)
- from 224 tweeters
- between 2018-10-07 07:03:22 and 2018-10-12 17:36:22 (Pacific/Auckland).
5 Analysis
5.1 Tweets and Tweeters over time
Figure 5.1: Number of tweets and tweeters
Figure 5.1 shows the number of tweets and tweeters in the data extract by day. The quotes, tweets and re-tweets have been separated.
If you are in New Zealand and you are wondering why there are no tweets today (2018-10-12) the answer is that twitter data (and these plots) are working in UTC and (y)our today() may not have started yet in UTC. Don’t worry, all the tweets are here - it’s just our old friend the timezone… :-)
5.2 Who’s tweeting?
Next we’ll try by screen name.
Figure 5.2: N tweets per day by screen name
Figure 5.2 is a really bad visualisation of all tweeters tweeting over time. Each row of pixels is a tweeter (the names are probably illegible) and a green dot indicates a few tweets in the given day while a red dot indicates a lot of tweets.
So let’s re-do that for the top 50 tweeters so we can see their tweetStreaks (tm)…
Top tweeters:
| screen_name | nTweets |
|---|---|
| AlecTang_ | 83 |
| Davidxvx | 61 |
| GoodSenseMktg | 57 |
| aidselva | 36 |
| EDS_Aotearoa | 24 |
| KatevP | 24 |
| ENVIR_HEALTH | 23 |
| E_MSolutions | 21 |
| abbie_reynolds | 21 |
| anarkaytie | 20 |
| SilverLiningGS | 19 |
| SustainableAkl | 19 |
| DanHikuroa | 19 |
| gauloir | 17 |
| sustbusiness | 15 |
And their tweetStreaks are shown in Figure 5.3…
Figure 5.3: N tweets per day by screen name (top 50, reverse alphabetical)
Any twitterBots…?
5.3 Which hashtags are mentioned the most?
This is very quick and dirty but… to calculate this we have to do a bit of string processing first.
This is how I have tidied the hashtags (make other suggestions here):
# First we make everything lower case
htLongDT <- htLongDT[, `:=`(htLower, tolower(htOrig))] # lower case
# Next we remove the macrons just in case h/t:
# https://twitter.com/Thoughtfulnz/status/1046685305569345536
htLongDT <- htLongDT[, `:=`(htClean, stringr::str_replace_all(htLower, "[āēīōū]",
myUtils::deMacron))]
# we might need to do other things here depending on the the contextTable 5.2 shows the total count of each #hashtag by (re)tweet type.
| hashTag | type | count |
|---|---|---|
| ccbc18 | Tweet | 320 |
| edsclimatechat | Tweet | 308 |
| ccbc18 | Re-tweet | 100 |
| edsclimatechat | Re-tweet | 58 |
| ccbc2018 | Tweet | 58 |
| climatechange | Tweet | 56 |
| climate | Re-tweet | 49 |
| climatechange | Re-tweet | 44 |
| climateaction | Tweet | 31 |
| climateaction | Re-tweet | 27 |
| climate | Tweet | 24 |
| ccbc18 | Quote | 18 |
| edsclimatechat | Quote | 16 |
| nz | Tweet | 15 |
| sr15 | Tweet | 12 |
| business | Re-tweet | 12 |
| climateleaderscoalition | Tweet | 11 |
| letsdothis | Re-tweet | 11 |
| ccbc2018 | Re-tweet | 9 |
| greenbond | Re-tweet | 9 |
Figure 5.4 plots the daily occurence of these hashtags after removing variants of #edsclimatechat OR #ccbc18 OR #ccbc2018 and selecting only those which have more than 10 mentions on any day. For clarity tweets and re-tweets are aggregated. See Section 7 for the problems with this #hashTag counting approach.
Figure 5.4: Most mentioned #hashtags per day (only > 10 per day shown)
6 Most popular hashtags over time
There are a lot of problems with this approach (see Section 7) but Figure 6.1 shows trends over time (watch for lines of apparently dis-similar hashtags where the macron fix has failed) and Figure 6.2 shows the totals to date.
Figure 6.1 uses plotly to avoid having to render a large legend - just hover over the lines to see who is who…
Figure 6.1: Cumulative hashtag counts over time (only total count >10 shown)
Figure 6.2: Total hashtag counts to date (only total count > 10 shown)
7 Problems
Loads of them. But primarily:
- hashTags as indicators:
- tweets mentioning a topic without using a hashtag will not show up so;
- there is no sentiment analysis;
- removing the macrons: the tables and plots may display incorrect Māori words. Unfortunately I have not found a way to put them back in the right words (and only the right words).
8 About
As ever, #YMMV.
Analysis completed in 9.54 seconds ( 0.16 minutes) using knitr in RStudio with R version 3.5.1 (2018-07-02) running on x86_64-apple-darwin15.6.0.
A special mention must go to https://github.com/mkearney/rtweet (Kearney 2018) for the twitter API interaction functions.
Other R packages used:
- base R - for the basics (R Core Team 2016)
- data.table - for fast (big) data handling (Dowle et al. 2015)
- ggplot2 - for slick graphs (Wickham 2009)
- plotly - for interactive graphs (Sievert et al. 2016)
- readr - for nice data loading (Wickham, Hester, and Francois 2016)
- reshape2 - wide to long (Wickham 2007)
- rtweet - twitter API search (Kearney 2018)
- stringr - processing strings (Wickham 2016)
- knitr, bookdown & kableExtra - to create this document (Xie 2016b,Xie (2016a), Zhu (2018))
References
Dowle, M, A Srinivasan, T Short, S Lianoglou with contributions from R Saporta, and E Antonyan. 2015. Data.table: Extension of Data.frame. https://CRAN.R-project.org/package=data.table.
Kearney, Michael W. 2018. Rtweet: Collecting Twitter Data. https://cran.r-project.org/package=rtweet.
R Core Team. 2016. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.
Sievert, Carson, Chris Parmer, Toby Hocking, Scott Chamberlain, Karthik Ram, Marianne Corvellec, and Pedro Despouy. 2016. Plotly: Create Interactive Web Graphics via ’Plotly.js’. https://CRAN.R-project.org/package=plotly.
Wickham, Hadley. 2007. “Reshaping Data with the reshape Package.” Journal of Statistical Software 21 (12): 1–20. http://www.jstatsoft.org/v21/i12/.
———. 2009. Ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. http://ggplot2.org.
———. 2016. Stringr: Simple, Consistent Wrappers for Common String Operations. https://CRAN.R-project.org/package=stringr.
Wickham, Hadley, Jim Hester, and Romain Francois. 2016. Readr: Read Tabular Data. https://CRAN.R-project.org/package=readr.
Xie, Yihui. 2016a. Bookdown: Authoring Books and Technical Documents with R Markdown. Boca Raton, Florida: Chapman; Hall/CRC. https://github.com/rstudio/bookdown.
———. 2016b. Knitr: A General-Purpose Package for Dynamic Report Generation in R. https://CRAN.R-project.org/package=knitr.
Zhu, Hao. 2018. KableExtra: Construct Complex Table with ’Kable’ and Pipe Syntax. https://CRAN.R-project.org/package=kableExtra.