Creating a Twitter Bot for a Rock Band

twitter-botBackground

I was approached recently by a band who wanted to promote their latest song on Twitter. They knew that a lot of people were talking about them online and they wanted to use that conversation to highlight their new single. This made perfect sense; people already were talking about them, what better time is there to highlight their new music?!

We discussed a tool whereby we would search for people who mentioned them in a tweet and respond to those users with a link to their latest song. This was an interesting task with some interesting challenges.

1. What tweets should we look for?
Any tweets containing the bands name or their twitter handle

2. What tweets should we reply to?
The tweets are validated to see if they are worth responding to:

  • Has the bot already tweeted this person?
  • Has the tweet got any negative words in it that we want to avoid?
  • Is the tweet a RT, and if so, do we want to ignore this?
  • Has this tweet been responded to before? If any of these are true, we should ignore the tweet.

Otherwise, we will respond with a choice of replies

  • Has the tweet got the name of the bands latest song in it?  Wouldn’t we want ensure we don’t send them back the same song? We would look like lunatics! Example:
    • Twitter User: I like Ice Cream!
    • Bot: Hey, have you heard about Ice Cream??!!!

3. Lets friend request the people tweeting about us!
This is useful as you can send a friend request which won’t execute if they are already a follower!

4. How will we record the data?
I use the pickle module here to record all tweets we respond to. This data informs future validation checks! Awesome…

5. When will we run it?
Daily of course! A borg file will execute the python file every night!

Summary

Finally, after a lot of testing, we had a working tool which would :

  1. Fetch x amount of tweets every day
  2. Decide whether it should reply
  3. Respond with a link to the song or not depending on the tweet content
  4. Record all data and use for future decisions

The band was very happy and I thoroughly enjoyed the process. Tweepy is an awesome Python library that makes querying the Twitter API painless. If you want to check out the code, head over to my github!

Advertisements

Switch Statement in Python

In the fantastic Import Python newsletter, (well worth signing up to by the way), I read the article Why Doesn’t Python Have Switch/Case? This is an interesting question that hadn’t crossed my mind, which goes to show that I haven’t actually missed it all that much. However, who doesn’t want another tool for their arsenal? Daniel’s solution of using the dict.get() function to return a default value is not only practical but also very elegant.

But what about a switch statement where the parameter isn’t exact? What if we want to return a value when the parameter is LIKE something?

I modified Daniel’s switch function to include a default ‘like’ argument which, when == True, will iterate over the keys of the dict looking to see if ‘arg’ is in the key. See code below :

This new switch function includes the following arguments:

  • arg (string) : The term we are searching for.
  • default_case (any type) : The value to return if nothing is found
  • like (bool) – optional : Whether we want to search for values where they are like the keys in the dictionary.

 

 

Assumptions: Case Sensitivity

If the user wants to look for values that are like x, they don’t want case sensitivity to get in the way of a positive result. We cater for this by invoking the lower() function on the arg and keys of the dictionary. We also parse these to strings to ensure that no errors are thrown if they keys are of other types.

Use Cases:
>>> switch("Monday", "Day does not exist")
1

>>> switch("Mon", "Day does not exist")
"Day does not exist"

>>> switch("Mon", "Day does not exist", like=True)
1

Finally: (little python pun)

I questioned whether it would be nicer to employ a lambda function in the except block rather than the 5 lines. However, for the sake of readability, I left it as is. I really enjoyed modding this code and hope it was interesting for you too! Until next time, happy coding…

Swift – Apples new flagship programming lanaguage

Image

On June 2nd, at Apple’s WWDC (World Wide Developer Conference), many exciting new changes were unveiled to the world. Apple shed light on what their bets are for the coming year(s).

While we got a glimpse of the new Mac OS Yosemite, and a sneak peek at iOS8, the real shock came at the unveiling of their new and unexpected programming language; Swift.

For those who don’t know a lot about programming languages, let me make this as easy as possible. Swift is a programming language that will allow users to build mobile applications and Mac OS applications in a much easier way. The language looks more like a scripting language (eg. JavaScript, Python, etc), rather than usual programming languages (C, Objective C, Java, etc).

Apple is putting their money behind developers. They want more apps developed, at a higher quality, at a quicker speed. To do so, they have equipped developers with a new programming language and a new development arena in which to do so (XCode 6). A week has gone by since Apple Developers have had a chance to get their hands on the product. The general opinion on the internet has been supportive and exited. I am yet to create my first swift iOS app, but it’s only a matter of days. I will keep you updated here!

Twitter opens Data Grants Programme

Twitter is arguably one of the best sources of opinion based data on the internet. It is awash with global sentiment reflecting upon countless topics and issues from entertainment to politics. However, the ability to mine historical Twitter data has been restricted in the past as data is Twitter’s currency and they obviously want to protect it.

However, in an exciting new development, they have created Data Grants which will connect “…research institutions and academics with the data they need.” These Data Grants allow for research of historical tweets (excluding deleted tweets and direct messages) to be collected and analysed by searching for hastags, keywords, date ranges and other variables. The only caveat is that the grant is only available for non commercial research purposes.

This is sure to have a deep impact on the research being carried out at academic institutions around the world. While the idea is still fresh and in its infancy, the possibilities of this research are endless: Virus Prediction, Signs of Social Unrest and no doubt some form of X-Factor Insights (for better or for worse!)

Find out more about Twitters Data Grant here:

https://engineering.twitter.com/research/data-grants

There’s An R Package For That

This short, witty video made by the smart guys at John Hopkins University parodies the famous Apple adverts. Instead of the well known phrase “There’s an App for that…”, we are treated to the notion that “There’s an R Package For That”. While this quick video hilariously mimics the music, language and tone of voice of an Apple advertisement, the beauty lies in the truth. There truly is an R package for the majority of things you are looking for.

Instead of reiterating what has been well written somewhere else, let me guide you to the fantastic breakdown of 2013’s 100 best R packages:
http://www.r-statistics.com/2013/06/top-100-r-packages-for-2013-jan-may/

Life, Death & Marriage – Ireland’s Last Fifty Years

The recent decades have seen a lot of change in Irish culture. Our lives have been disrupted by recession, enlightened by the Internet and changed dramatically due to political , social and health reformation. But how, if anything, have the past few decades affected our birth, death and marriage rates? Has anything changed? Has everything changed?

I pulled down some raw data from the Central Statistics Office of Ireland and decided to look at the trends.

*All data clean up was done in Microsoft Excel, then imported to R for Statistical Analysis where it was graphed.

Birth Rates:

Registered Births Per Quarter in ireland 1960 - 2012

The graph above plots a blue circle for each quarter of the years 1960 – 2012. Each blue circle represents the total registered births for that quarter in Ireland and the blue trend line displays the average birth rate.

  • From 1960 until 1977, birth rates are on the incline.
  • At the start of the 1980s, registered births start to decline drastically and continue to do so until 1994.
  • Potential factors: 1980s recession and mass emigration surely played a big role.
  • Interestingly, since ’94, birth rates are once again on the incline and show no immediate sign of slowing down.
  • On average, there are 15.6k births per quarter.

Marriage Rates:

Registered Marriages Per Quarter in ireland 1960 - 2012

The Registered Marriages graph plots the total amount of marriages per quarter in Ireland. Each purple dot represents the total amount of marriages for that quarter.

  • You will notice that there seems to be three separate lines of dots. These dots correspond to different activity per quarter.
  • The higher line of dots is due to the high amount of marriages that take place during Quarter 3 of each year (Warm summer months, lovely!).
  • Q3  accounts for 39% on average of all Irish marriages during a year.
  • The lower line of dots appears to start around 1984 and accounts for all  Q1 marriages (Cold winter months, your big day will be ruined with rain and snow!).
  • Q1 accounts for 19% on average of all Irish marriages during the year.
  • On average, there are 4.8k registered marriages per quarter.

Marriages also took a small dent during the recession of the 80s but is climbing back up again. It would be interesting to graph the marriage rates against recorded religious beliefs to see how they have changed over the years and if this has had any affect on marriages… If I can get this data, expect some ground breaking insights (no guarantee of any ground being broken by these insights)

Death Rates:

Registered Deaths Per Quarter in ireland 1960 - 2012

Finally, I know you are all excited to see how we are doing in the fight against death! You’ll be happy to hear (I hope) that the amount of registered deaths in Ireland is also on the decline.

  • The higher quarterly death rates appear in Q1 of most years with 29% of the total yearly deaths taking place in that quarter.
  • Q2, Q3 and Q4 represent 25%, 22% and 24% of yearly deaths.

Now to truly understand this data, we need to understand our population growth over the past 50 years. Ireland has grown from a recorded population of 2.8 million in 1961 to 4.59 million in 2011. Thats a 62.8% increase.

So what have we learned?

  • Birth rates are increasing again and don’t seem scared of this latest recession!
  • For every marriage per quarter, we pop out 3.3 kids!
  • We are afraid of being cold/soaked at our weddings!
  • We are dying at a decreasing rate, more time for Werther’s Originals when we are older!

R Simplified: Playing with Dataframes (1)

Fantastic blog post on using R for data analysts with SQL experience!

vasaptexblog

R is well established language in the community and it has a huge fan following.  Big Data being the buzzword of today, more and more people are learning this to add predictive flavor to their data analysis. I believe, learning becomes easier if one is able to relate new things with something which he/she already knows. So assuming that my hypothesis is correct, in this blog, I have discussed R statements/commands corresponding to SQL- Data Manipulation Language (DML).

We may find these if we search internet. However in this blog we will find relevant R commands consolidated at one place along with corresponding SQL statements so that we can relate. As R has a long list of packages so in case of a particular package, syntax may be different.

These commands are from base R package. I am using Rx64 3.0.2 in windows environment. To test the R command discussed…

View original post 340 more words