Using Social Media for Research, Monitoring and Evaluation in the MENA Region: World Food Programme Case Study

"This report presents an evaluation of the utility of social media for conducting research, monitoring and evaluation in the MENA region, using a case study approach."

From the Middle East and North Africa (MENA) Division, Department for International Development (DfID) United Kingdom, this evaluation report builds ion the DfID Practice Note: Using Social Media Data in International Development Research, Monitoring and Evaluation by using the methods described there to study a Twitter-based reaction to the reported cancellation of a World Food Programme (WFP) food provision programme in the Mena region. Because big data is readily available from social media (used in the MENA region by 85% of the population on Facebook and 45% on Twitter), opportunities for insights are available to researchers.

"The following questions were addressed via an analysis of the tweet dataset: 

• Do the spikes in frequency of tweets collected with the key term set correspond to WFP press releases and Google Trends? 
• Which social media users are most influential in spreading information about the WFP cuts? 
• Can we identify clusters of geo-located data in different regions? 
• How does sentiment change over time? 
• Are social media posts containing negative sentiment in relation to the WFP cuts more likely to spread compared to posts containing positive sentiment?"

Methods used include: frequency analysis, topic classification; social network analysis, geospatial analysis, sentiment analysis, and information propagation modelling. The use of big data alone has limitations, for example, population demographic characteristics cannot be obtained. However, the near real-time aspect of big data from social media lends itself to major events or incidents such as civil unrest or foreign policy interventions, for example.

Frequency (of tweets or of a keyword in tweets) analysis "allows a researcher to visually identify points of high and low activity in relation to an event or topic...." For example, “the case study showed that information regarding ebb and flow of the popularity or awareness of the cuts to the WFP could be obtained from Twitter.  Spikes in popularity or awareness on Twitter related to key reports in the media relating to the cuts. These spikes were subjected to further analysis using topic classification."

Topic classification of the tweets utilised algorithms to reduce textual information into summaries of the main themes shown as wordclouds. "Words are sized in the wordcloud based on their frequency - the bigger the word, the more frequently it is used.  The accounts @60Minutes and @bbcworld were dominant in the visualizations, confirming the broadcast reaction on Twitter."

Social Network Analysis is used to visualise interactions of social media users through social networks by highlighting prominent user accounts in the dataset on a graph, for example, between accounts posting about cuts to the WFP on Twitter.  "...The graphs also identified nodes (accounts) that acted as bridges for information to travel from one Twitter community to another (the removal of these accounts would have the impact of stemming the flow of information between these communities)."   

Geospatial Analysis comes from various sources. User supplied data has the limitation of users not supplying information or supplying incorrect information. However, it is "possible to identify the country for over 50% of Twitter users....The Yahoo! PlaceFinder geographic database can be used to extract location information....Geotagged data tell researchers where a person is when they publish the tweet, whilst profile data could tell researchers any number of things including where people were born, lived, employed, are passing through or simply identify with." Twitter users may opt in to have tweets geotagged for location: "...approximately 1% of all tweets are geotagged globally", making the dataset very small in a particular region." Language use can be an additional statistic available for analysis of tweets.

"Sentiment analysis provided insights into the emotive response of Twitter users to the cuts to the WFP.... The outcome of sentiment analysis is often subjective and based on the existence of a list of keywords in a message." However, validation of the "ground truth" of sentimental analysis tools is done by comparing human users sentiment scoring with algorithm results. The algorithms are geared to measure positive and negative content of tweets and are not sensitive to such things as sarcasm. "The WFP Twitter dataset showed that there were more extreme negative views than extreme positive views, and that peaks in negative views corresponded to the press releases during the study window." 

Through information propagation statistical modelling, "it is possible to build statistical models of the flow of information on social media networks to gain better insights into the dynamics of propagation...the spread of social media messages."  This model is an analysis of the size of information (measured by counting number of retweets - volume of public interest and endorsement of content) and survival/of information (measured by counting the seconds between the first and last retweet - persistence of information over time). "For example, if it is identified that certain types of accounts are important to the  spread of information (e.g. local MENA agents) then it would be possible to seek  endorsement of key MENAD messages by these agents, via retweeting."

The report discusses contention in the area of ethics, the introduction of ethical principles, legal consideration, public attitudes, and publication of quantitative and qualitative findings. "Key issues include informed consent, confidentiality and potential harm to social media users.  The emerging consensus from government and  academia, is that while it is acceptable to collect and analyse social media posts  without consent from users, informed consent should be sought (opt]in or opt]out) if  posts are to be directly quoted in research reports."