We Feel

We Feel is a project that explores whether social media - specifically Twitter - can provide an accurate, real-time signal of the world's emotional state.

Why did you make it?

Hundreds of millions of tweets are posted every day. A huge topic of conversation is, of course, the authors; what they are up to, what they have encountered, and how they feel about it.

We Feel is about tapping that signal to better understand the prevalence and drivers of emotions. We hope it can uncover, for example, where people are most at risk of depression and how the mood and emotions of an area/region fluctuate over time. It could also help understand questions such as how strongly our emotions depend on social, economic and environmental factors such as the weather, time of day, day of the week, news of a major disaster or a downturn in the economy.
How do you obtain the tweets?

We Feel has three sources of tweets: a random 1% sample from the public Twitter API (affectionately known as the gardenhose), a random 10% sample from GNIP (called the decahose), and a third source that specifically monitors the public Twitter API for a large vocabulary of emotion terms (we call this the emohose).
What volume of tweets do you handle?

The gardenhose delivers about 900 thousand English tweets a day, of which about 250 thousand contain emotion terms. The decahose predictably delivers ten times that amount. The emohose delivers about 27 million English tweets a day, all of which contain emotion terms. That averages out to 19 thousand tweets per minute, though this fluctuates quite wildly.
How do you identify emotions?

We Feel uses a large vocabulary of emotion terms that were compiled from multiple sources, including the ANEW and LIWC corpora, and a list of moods from LiveJournal. We conducted a crowdsourcing task (using Crowdflower) to organise these terms against Parrott's hierarchy of emotions. The emotions are colour-coded using a dataset of affective norms provided by the Center for Reading Research at Ghent University.
How do you identify gender?

We determine gender by inspecting the user name of a Twitter account, and comparing it against large lists of male and female names obtained from the BabyNameMap project.
What's up with these weird maps?

Unfortunately, it isn't easy to plot Twitter accounts on a map. People can state their location in their Twitter profile, but they do this in plain text. They can just as easily say "the moon" or "none of your business", or certian celebrities' bedrooms. It isn't currently practical for us to resolve these texts for the millions of tweets we encounter every hour.

So, we use time zones to roughly locate accounts. That works fine in Australia (which is our focus), where they correspond nicely to state boundaries. It doesn't work so well in places like North America. This is something we would like to work on in the future.
How do you handle the tweets?

We rely heavily on the Amazon AWS infrastructure to deal with this volume of data. We use several EC2 instances to capture tweets, annotate them (with emotions, gender and rough location) and build up the summaries that are needed by the visualisation.

Communication between these instances is achieved with Kinesis: a simple yet highly scalable pipeline through which we funnel our tweets and annotations. This allows anything consuming the data to be freely restarted and rescaled without missing a beat (or tweet).

The tweets are backed up to S3 for cold storage, and a summarised into DynamoDB tables to be queried by the web application. We also use Cloudwatch to monitor everything.

These excellent frameworks have allowed us to construct a highly responsive web application that should be able to run on any modern browser, without any annoying dependencies like Flash or Java.