class: center, middle, inverse, title-slide # rtweet for Rfun ## Packages to gather and analyse tweets ### John Little ### 2017-03-27 --- exclude: true ## Setup for the GraphTweet slides --- class: center, bottom background-image: url(http://library.duke.edu/data/sites/default/files/datagis/images/data_gis_logo.png) #[is.gd/rfunsurvey2017](https://is.gd/rfunsurvey2017) --- class: center, middle # Presentation Materials [Slides](//libjohn.github.io/rtweet/slides.html) [Github](//github.com/libjohn/Rfun/tree/master/rtweet) --- class: middle, center ## Twitter Stream Gathering [rtweet](https://github.com/mkearney/rtweet/) [twitteR](https://cran.r-project.org/web/packages/twitteR/) --- class: middle, center background-image: url(images/twitter-analysis.png) <!-- https://commons.wikimedia.org/wiki/File:NodeXL_Twitter_Network_Graphs_-_Occupywallstreet_(mentions_and_replies)_(BY).png --> .right-column[ <h2>Twitter Analysis</h2> ] --- ## Outline - Using rtweet (a tidy way) - library(twitter) should also work fine - Analysis demonstrations - WordCloud - Word Freq - term document matrix - Sentiment Analysis - Network Graphs / Network Analysis - Time Series - Streaming / Scheduling / Sampling --- class: middle, center, softblue ## Gettting Started [Intro Vignette](https://cran.r-project.org/web/packages/rtweet/vignettes/intro.html) --- ## Authentication - API - https://apps.twitter.com/ - Keys and Access Tokens - Must be careful with the secret code - https://mkearney.github.io/rtweet/articles/auth.html - But, my examples don't seem to require keys --- ## Gathering is Easy ```{} mm_tweets <- search_tweets("marchmadness", n=1000, lang = "en") ``` ```{} users_data(mm_tweets) ``` There are significant limitations in gathering historical data from the twitter API --- ## API Orchestration Many Tools can gather - Easy: https://tags.hawksey.info/ - R with rtweet - Splunk --- class: softblue ## Analysis 1. Word Cloud 2. Sentiment Analysis 3. Network Graph Analysis 4. Time Series --- class: orange ## Word Cloud - WordCloud2 (HTML Widget) - Requires treatment and transformations - Term Document Matrix - Text Mining - lower case - strip whitespace - remove punctuation - remove numbers - remove stop words - term stemming --- class: bottom, center background-image: url(images/word-cloud.png) [Example 1](rtweet4rfun.nb.html): Search Tweets | Data Treatment | TDM | WorldCloud --- class: orange ## Sentiment Analysis Applied a simpler text treatment for this demonstration. See [Example 1](rtweet4rfun.nb.html) for a more complete treatment of data cleaning. ```{} iconv(TweetText, 'UTF-8', 'ASCII') -> UseableText ``` Get sentiment ```{} get_nrc_sentiment(UsableText) ``` --- class: bottom, center background-image: url(images/sentiment_vis.png) [Example 2](sentiment_analysis.nb.html): syuzhet::get_nrc_sentiment(), plot sentiment --- class: orange ## Network Graph - `library(graphTweets)` - Transforms the document into edges and nodes - Creates a Gephi Document: `graphTweets.graphml` &nbsp; &nbsp; - HTML Widget [DiagrammeR](https://github.com/rich-iannone/DiagrammeR) is worth investigation --- class: bottom background-image: url(images/network_graph.png) 1. [Example 3](network_graph.nb.html): `getEdges()` | `getNodes()` | plot 2. Launch Gephi > Open Graph File > graphTweets.graphml --- class: center, middle ## More graphTweets Examples For the next few slides see [my R Notebook](network_graph_more_examples.nb.html) for code details --- class: middle ### graphTweets -- Identify Edges <!-- --> ``` dukmbb <- search_tweets("dukembb", n=100, lang = "en") ``` --- ### Edges with NetworkD3 <div id="htmlwidget-b72c2538687d73a662a3" style="width:504px;height:504px;" class="forceNetwork html-widget"></div> <script type="application/json" data-for="htmlwidget-b72c2538687d73a662a3">{"x":{"links":{"source":[36,78,92,92,114,114,114,114,114,27,10,64,64,135,6,6,6,6,6,128,128,128,128,65,96,96,96,75,75,49,68,80,134,134,134,54,54,54,54,2,2,2,2,2,71,97,138,139,21,130,130,130,130,98,40,94,69,69,69,39,133,33,136,136,136,136,136,55,55,55,55,55,136,136,117,57,57,57,57,57,57,45,45,110,110,110,110,110,19,111,111,111,108,108,108,122,109,84,84,124,124,106,106,106,106,106,131,93,99,59,59,59,113,113,113,113,113,113,113,113,113,50,50,50,50,50,53,53,53,35,35,35,35,35,123,123,123,123,13,5,5,56,56,56,91,91,91,24,24,18,95,95,95,18,101,101,101,107,107,107,88,88,132,132,121,121,121,15,15,15,15,15,137,74,74,74,74,74,120,120,120,15,41,41,41,41,41,89,89,89,89,89,58,58,44,44,47,47,90],"target":[29,29,12,29,82,81,29,23,102,29,29,17,77,29,63,86,7,29,62,96,16,67,28,29,16,67,28,51,31,29,29,29,25,26,29,29,60,73,126,82,81,29,23,102,29,29,29,29,29,95,9,29,87,29,29,29,103,29,127,29,29,29,55,43,60,37,29,136,43,60,37,29,38,30,29,88,101,29,60,61,126,48,29,82,81,29,23,102,29,79,29,129,84,29,126,29,29,29,126,29,66,1,127,29,14,125,29,29,29,112,29,60,3,60,51,34,37,20,76,43,29,1,127,29,14,125,29,126,4,82,81,29,23,102,83,115,32,29,29,8,29,91,29,11,56,29,11,116,29,29,9,29,87,66,88,70,0,88,70,0,100,22,46,29,126,42,29,74,119,29,52,104,29,15,119,29,52,104,118,85,29,105,1,127,29,14,125,1,127,29,14,125,48,29,72,29,48,29,29],"value":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],"colour":["#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666","#666"]},"nodes":{"name":["","_JOakes_","_Joay32","accmbb","accmbb?","akeeferlex","alliseeisgold24","APlayersProgram","ariana2525","ArmyWP_MBB","asiliybaswgj","BadgerMBB","BarnhillDerek","barrier_nathan","BaylorMBB","blasianhoe","bluedevilsvoice","c393k","CameronDick27","Cantbebeat_JC12","courierjournal","D6T7xduOYDO6ray","D9MAXTu0rk","DallasMavs","DaMastaAssassin","dcip23","DickieV","DominicVirelli","dukeMBB","DukeMBB","DukeMBB!","DukeMBB.","DukeU","ekaterina2350","ESPNCBB","Estephane21","Feverz_","FinalFour","FinalFour.","florenzaoliv","GerminiyLapin","GhantM","GoCards","GoHeels","GoseyHannah","GreatDBates","GTMBB","HapsterNC","IHateDuke1","iriabuki","jasonhanse","JayBilas","jaytatum0","jimmy0726","JMikeMozingo","jools6691","JoseJacobo16","kaylasmedley_","keenun54","KeithMcSheaBNG","KentuckyMBB","KUAthletics","KUHoop","KUHoops","kulagingektor3","kushpatel28","KyrieIrving!!","laettnerbball","lanagru2","lannodisha1973","letics…","Lucythemoon2","LukeCombsMusic","macwfnz","masontyler908","MathiasB26","Maybin939","MBB htt","melindatro","MickyArison","MommaElyse","NBA","nbadleague","NCAA","Nemirjaafar44","NOTSportsCenter","NovaMBB","OregonMBB","Paolo1123","Peter42Kerley","PopularHustler","QJones_2","raisaayxkor","ranababar1988","RfoqBipu8AYaP73","richdemarco","rmccoy1963","robintimsz","ruslanuaden","RussellKriz","s//t","samuelzelalem","sdotcu…","Simpson16Tanner","sixers","Sixers","sparkky1919","spencer_vela","SportsRcollege","Stayingoutheway","Tagz_1","tarheelinavy","TBNSports","TDinsight","that_dude_ezy","theACC","TheHerd","TheRealPhara0h","TheSportsPoller","theStoryofArose","TimS70","TomWimsatt2","Trenton_stg","two2eagles","TylerBurggraff","UF","UNC_Basketball","UofSC","VoicesWife","WayneElli22","West_Point_Fans","WLA_hoops","woodysmalls","worltachiza1974","yakovlevaolimp6","yaromirakoshel6","yesnicksearcy","yoshi_26_nari","Zaphnath_P","zymg8YcvKZacFNA"],"group":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],"nodesize":[8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8]},"options":{"NodeID":"name","Group":"group","colourScale":"d3.scaleOrdinal(['#3182bd'])","fontSize":7,"fontFamily":"serif","clickTextSize":17.5,"linkDistance":50,"linkWidth":"'1.5px'.toString()","charge":-30,"opacity":0.6,"zoom":false,"legend":false,"arrows":false,"nodesize":true,"radiusCalculation":"d.nodesize","bounded":false,"opacityNoHover":1,"clickAction":null}},"evals":[],"jsHooks":[]}</script> --- ### Nodes plot from [Coene's example 2](http://john-coene.com/packages/graphTweets/examples#example_02) <!-- --> --- ### Geocode tweeters with ggmap and leaflet <div id="htmlwidget-a0d2aa60b03ade92e67c" style="width:504px;height:504px;" class="leaflet html-widget"></div> <script type="application/json" data-for="htmlwidget-a0d2aa60b03ade92e67c">{"x":{"options":{"crs":{"crsClass":"L.CRS.EPSG3857","code":null,"proj4def":null,"projectedBounds":null,"options":{}}},"calls":[{"method":"addTiles","args":["https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png",null,null,{"minZoom":0,"maxZoom":18,"maxNativeZoom":null,"tileSize":256,"subdomains":"abc","errorTileUrl":"","tms":false,"continuousWorld":false,"noWrap":false,"zoomOffset":0,"zoomReverse":false,"opacity":1,"zIndex":null,"unloadInvisibleTiles":null,"updateWhenIdle":null,"detectRetina":false,"reuseTiles":false,"attribution":"© <a href=\"http://openstreetmap.org\">OpenStreetMap<\/a> contributors, <a href=\"http://creativecommons.org/licenses/by-sa/2.0/\">CC-BY-SA<\/a>"}]},{"method":"addMarkers","args":[[35.410244,35.462187,31.25044,35.462187,43.41242,35.850295,25.753332,52.52437,47.45371,52.52437,47.45371,36.396597,36.396597,36.301068,35.26002,37.323275,36.301068,55.75222,35.26002,52.16045,55.75222,37.323275,38.91706,-18.4746,36.165688,38.985932,35.633598,35.96099,35.162228,38.895112,34.345106,33.5091667,42.89606,32.781179,33.987454,45.371478,32.931523,36.396597,26.71037,38.69377,26.71037,26.71037,36.625349,36.625349,36.625349,36.625349,41.107856,36.625349,36.625349,36.625349,36.625349,36.625349,36.625349,33.952602,33.981735,35.26002,35.410244,36.396597,36.835426,35.7721,33.14874],[-81.202959,-80.898668,-99.25061,-80.898668,-83.886873,-79.023205,-80.270379,13.41053,-122.321791,13.41053,-122.321791,-78.976099,-78.976099,-115.206969,-80.804151,-121.931623,-115.206969,37.61556,-80.804151,-0.70312,37.61556,-121.931623,-77.00025,-70.29792,-86.778098,-76.744413,-79.852391,-78.25361,-96.609139,-77.036366,-80.073953,-111.8983333,-78.886424,-96.790329,-81.024864,-122.874613,-96.615821,-78.976099,-108.32446,-122.02299,-108.32446,-108.32446,-88.279562,-88.279562,-88.279562,-88.279562,-74.102352,-88.279562,-88.279562,-88.279562,-88.279562,-88.279562,-88.279562,-84.549933,-91.5578,-80.804151,-81.202959,-78.976099,-76.298274,-78.63861,-94.15019],null,null,null,{"clickable":true,"draggable":false,"keyboard":true,"title":"","alt":"","zIndexOffset":0,"opacity":1,"riseOnHover":false,"riseOffset":250},["Feverz_","melindatro","that_dude_ezy","melindatro","alliseeisgold24","VoicesWife","MathiasB26","iriabuki","lanagru2","iriabuki","lanagru2","JMikeMozingo","JMikeMozingo","_Joay32","Lucythemoon2","robintimsz","West_Point_Fans","florenzaoliv","Lucythemoon2","ekaterina2350","robintimsz","jools6691","ekaterina2350","florenzaoliv","TheRealPhara0h","kaylasmedley_","GreatDBates","Tagz_1","Cantbebeat_JC12","tarheelinavy","Trenton_stg","Stayingoutheway","sparkky1919","RussellKriz","KeithMcSheaBNG","TDinsight","jasonhanse","jimmy0726","Estephane21","barrier_nathan","JoseJacobo16","QJones_2","JoseJacobo16","JoseJacobo16","DaMastaAssassin","CameronDick27","CameronDick27","CameronDick27","CameronDick27","richdemarco","CameronDick27","CameronDick27","CameronDick27","CameronDick27","CameronDick27","CameronDick27","woodysmalls","yoshi_26_nari","masontyler908","TimS70","Peter42Kerley","keenun54","JMikeMozingo","GoseyHannah","HapsterNC","PopularHustler"],null,null,null,null,null,null]}],"setView":[[39.81039,-98.556102],3,[]],"limits":{"lat":[-18.4746,55.75222],"lng":[-122.874613,37.61556]}},"evals":[],"jsHooks":[]}</script> --- ### Notes on previous slide Although rtweet has `lookup_coords()` I did not find it to be successful for my test. It may work fine upon further review. I chose to use ggmap and leaflet. see [my R Notebook](network_graph_more_examples.nb.html) for code details --- class: orange ## Time Series https://cran.r-project.org/web/packages/rtweet/vignettes/stream.html --- class: softblue ## Other ### Scheduling - See the [time-series](https://cran.r-project.org/web/packages/rtweet/vignettes/stream.html) vignette - [taskscheduleR: schedule R scripts with Windows task manager](https://www.r-bloggers.com/taskscheduler-r-package-to-schedule-r-scripts-with-the-windows-task-manager-2/) ### Analysis & Issues - Machine Learning - Implications for Social Science ### Keeping up from a tool perspective - https://www.r-bloggers.com/search/Twitter/ --- class: center, bottom background-image: url(http://library.duke.edu/data/sites/default/files/datagis/images/data_gis_logo.png) #[is.gd/rfunsurvey2017](https://is.gd/rfunsurvey2017) --- ## Thank You For Attending .pull-left[ ### I am ... - John Little - http://libjohn.github.io - http://github.com/libjohn #### Schedule Me - [http:&#47;&#47;v.gd&#47;littleconsult](http://duke.libcal.com/appointment/2695) ] .pull-right[ ### We are... - Data & Visualization Services - http://library.duke.edu/data - The /Edge, Bostock (1st Floor) #### Walk-in Hours - [Schedule](http://library.duke.edu/data/about/schedule) #### Our Workshops - [Current Workshops](http://library.duke.edu/data/news) - [Past Workshops](http://library.duke.edu/data/news/past-workshops) #### Contact Us - askData@Duke.edu ] --- class: center, middle ## Shareable under CC BY-NC license Data, presentation, and handouts are shareable under [CC BY-NC license](https://creativecommons.org/licenses/by-nc/4.0/)