Friday 18 June 2010

How do you spell goal?

It's not often I get the chance to pursue three of my passions - sport, mathematics and online social media - at the same time. The 2010 Football World Cup combines these facets of life in a way we probably haven't seen before, providing numerous opportunities for data mining, funky visualisations and general nerd-indulgence, as well as knocking out twitter for a time.

One creative exploration I saw recently was by @neilkod, who collected data from 30 GB of tweets on how people spelt the word "goal". Data mining Twitter is an evolving field - see our recent story on how by using Twitter data you can predict the success of a film. You can find the full goal data table here, and I have listed the top and bottom few below:

Rank
Word
Count
1
goal
50225
2
Goal
11727
3
GOAL
4202
4
goAl
798
5
goall
340
6
GOAAAL
92
7
goaL
88
8
Goall
75
9
GoaL
69
10
GOAl
66
11
goaaaal
61
12
GOal
50
....
....
....
1249
GGGGGGGGGGGGGGGGOOOOO
OOOOOOOOOOOOOOOOOOOA
AAAAAAAAAAAALLLLLLLLLLLLLLLL
1
1250
GGGGGGGGGGGGGGGGGGGOO
OOOOOOOOOOOOOOOOAAAA
AAAAAAAAAALLLLLLLLLLLLLLLLL
1
1251
GGGGGGGGGGGGGGGGGGGOO
OOOOOOOOOOOOOALLLLLLLL
1
1252
GGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGOOOOOOOOOOO
OOOOOOOOOOOOOOOOOOOO
AAAAAAAAAAAAAAAAAAAAAAA
AAALLLLLLLLLLLLLLLLLLLLLLLLL
1

As expected, on top is the word "goal" (71%) followed by "Goal" (17%) and "GOAL" (6%). Then there are various misspellings, before the excited tweets come in, including the 140 character "Goooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooal".

Interestingly, the caps lock induced "gOAL" is only used once.

You can also visualise this in a chart - note that in the chart below, the x-axis is a log scale due to the fact that the leading terms are so far ahead:

Distribution of the word "goal"

Now it's time to get our nerd on....

Zipf's law is a curious law that arose out of an analysis of language by linguist George Kingsley Zipf, who theorised that given a large body of language (that is, a long book), the frequency of each word is close to inversely proportional to its rank in the frequency table. That is:


where a is close to 1. This is known as a "power law" and suggests that the most frequent word will occur approximately twice as often as the second most frequent word, which occurs twice as often as the fourth most frequent word, etc. There is an excellent take on this over at Plus Magazine. As you can see in the log/log chart below, after the 5th version of "goal" a Zipf curve fits remarkably well.

Distribution of the word "goal" - log/log chart

There has never been a real explanation of why Zipf's law should apply to languages and there is controversy surrounding whether it gives any meaningful insight. Power laws relating rank to frequency have been demonstrated to occur naturally in many places - the size of cities, the number of hits on websites, the magnitude of earthquakes and the diameters of moon craters have all been shown to follow power laws.

Wentian Li demonstrated in his paper Random Texts Exhibit Zipf's-Law-Like Word Frequency Distribution, published in IEEE Transactions on Information Theory, that words generated by simply randomly combining letters fit the Zipf distribution. Li showed mathematically that the power law distribution of frequency against rank is a natural consequence of the word length distribution, with words of length 1 occurring more frequently than words of length 2 and so forth. His underlying theory is that the rank distribution arises naturally out of the fact that word length plays a part - long words tend not to be very common, whilst shorter words are. Li argues that as Zipf distributions arise in randomly-generated texts with no linguistic structure, the law may be a statistical artifact rather than a meaningful linguistic property.

Our results mirror Li's quite closely. It is clear that the most used versions of the word "goal" - where goal is spelt correctly with various capitalisations - should not fit the Zipf distribution as these words are not random - they are the actual correct spellings people are looking to write in their tweets. However, after the initial few words, random spelling errors, and then the simple randomness of how long people hold their fingers on the keys in the excitement of a goal, take hold. From this point, we see exactly the same as Li - that the Zipf distribution arises from random words, with longer words less common than shorter words.

If you would like to see some truly insightful twitter world cup visualisations, check out The Guardian's World Cup 2010 Twitter replay. I haven't been replaying the Australia vs. Germany game very often....

57 comments:

  1. I have read your blog its very attractive and impressive. I like it your blog.

    Guest posting sites
    Education

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. Great article and it is very inspiring to me. I glad to satisfy with your good job. You put the effort is very superb and I always appreciate your unique information. Keep it up.
    Pega Course in Chennai
    Pega Course
    Job Openings in Chennai
    Placement in Chennai
    Oracle DBA Training in Chennai
    Linux Training in Chennai
    JMeter Training in Chennai
    Pega Training in T Nagar
    Pega Training in Anna Nagar

    ReplyDelete
  4. very interesting post.this is my first time visit here.i found so many interesting stuff in your blog especially its discussion..thanks for the post! http://199.192.25.192/

    ReplyDelete
  5. GbWhatsapp is the earliest mod of WhatsApp that’s typically updated up until today. It’s one more impressively customized version of WhatsApp GBWhatsApp can be like OGWhatsApp, but it has lots of boosted functions. You do not need to uninstall WhatsApp if you want to install GBwhatsapp apk. GBWhatsApp has excellent attributes; you can review under

    ReplyDelete
  6. famous jewellery shops in chennai
    The craze on jewelry never goes down. Are you looking for the best Jewellery shops in Chennai? Here, is the list for you.

    ReplyDelete
  7. will help us develop a comprehensive plan for protecting the critical Salesforce data. lightning interview questions 2021

    ReplyDelete


  8. Hey friend, it is very well written article, thank you for the valuable and useful information you provide in this post. Keep up the good work! FYI, please check these depression, stress and anxiety related articles.
    How to Build a Portfolio with ETFs, My vision for India in 2047 postcard, Essay on Unsung Heroes of Freedom Struggle

    ReplyDelete
  9. This post is so interactive and informative.keep update more information...
    Web Designing Course in T Nagar
    Web Designing Course in chennai

    ReplyDelete
  10. Clicksudd vă oferă https://clicksudd.com/clicksud/ cu cel mai recent Clicksud episodul video în HD. Vă permite să jucați și să vă bucurați de divertisment de la .

    ReplyDelete
  11. mira aquí series turcas online gratis con subtítulos en español https://seriesturcas.cc/ .

    ReplyDelete
  12. https://animeid.me ltimate entertainment platform for online series and online movies. Digital entertainment around the world is covered by and we are heading to one of the best online platforms in the world. understands the needs of the public and offers exactly what the visitors have, which makes them unique to us. I hope you will return to us so that we can continue like this.

    ReplyDelete
  13. https://animeyt2.net is the ultimate entertainment platform for online series and online

    ReplyDelete
  14. I'm very roused alongside your composing abilities as perfectly similarly as with the organization in your blog. Is that a paid subject or did you modify it yourself? At any rate stay up the superb quality composition, it's interesting to take a gander at an incredible blog like this one today.. https://beunnaomua.doodlekit.com/blog/entry/22608947/10-tips-for-making-a-good-courses-on-facebook-marketing-even-better

    ReplyDelete
  15. This is moreover a by and large brilliant post which I genuinely thoroughly enjoyed scrutinizing. It isn't every day that I have the probability to see something like this.. http://marcokrta829.image-perth.org/15-things-your-boss-wishes-you-knew-about-email-marketing-b2b

    ReplyDelete
  16. Thank you for providing us with such a pleasant and informative blog. Ziyyara’s expert-led platform offers the finest online tuition for English in Saudi Arabia, specifically designed to cater to the needs of learners in Riyadh and beyond.
    For more info visit Online Tuition for english in Saudi Arabia or Call +971505593798, +919654271931

    ReplyDelete
  17. This blog is genuinely impressive in all aspects, Thanks for sharing. If you've been struggling with reading comprehension, lack of confidence, or motivation and engagement in your language learning journey, Ziyyara Edutech has the perfect solution for you.
    For more info visit Spoken english language Class in kuwait Fahaheel

    ReplyDelete
  18. I'm glad to hear that you enjoyed the article! Unfortunately, as an AI language model, I don't have the capability to send emails or subscribe users to updates. However, you can bookmark the website or subscribe to the website's newsletter, if available, to receive notifications about new articles or updates.
    Look for security firms in New York City that specialize in providing personal security services. Consider factors such as reputation, experience, and client testimonials when evaluating potential firms.https://dahlcore.com/personal-security

    ReplyDelete
  19. Orbilogin.net is the gateway to managing your Orbi router, allowing you to customize settings, monitor connections, and update firmware for optimal performance. For additional assistance, visit our site or consult our technical experts via live chat.

    ReplyDelete
  20. Effortlessly manage your Linksys extender settings with the Linksys extender login. For more assistance, visit our site or get expert help through live chat with our technical experts.

    ReplyDelete
  21. Initiating your printer setup through ij.canon.start is straightforward and user-friendly, ensuring a seamless installation process. For additional assistance, visit our site or connect with our technical experts via live chat.

    ReplyDelete
  22. Laview security camera setup is a straightforward process. Follow the user manual for a seamless installation experience. For additional assistance, visit our site or engage with our technical experts via live chat.




    ReplyDelete
  23. This article is a gem! The lorex home login information is like a secret weapon for hassle-free home security. Hats off to the Lorex Login service team for their dedication to simplifying the process and providing unmatched support.

    ReplyDelete
  24. Thanks a bunch for the fantastic tip! The irobot roomba setup setup has transformed the way I clean my home. Following their guide was a breeze, and I'm now enjoying a cleaner space with minimal effort. Appreciate the recommendation!

    ReplyDelete
  25. I'm truly impressed by the effort put into crafting the how to reset zmodo camera guide. It's rare to find such a comprehensive yet simple resource in the realm of home security. Thank you for sharing this gem!

    ReplyDelete