Some Twitter Infographics

I did some stuff like this before. And I figured, while I was updating my network graphs, why not update some of the other graphics?

And it helps that I worked out how to easily extract data from Twitter (see previous blog). The code is here. Again, rate limits apply.


Who Do I Follow?

This is one of the ones I did before - collect together the bios of the people I follow, then make a word cloud (using Wordle)

Basically, I follow a bunch of geeks and writers. Who like 'things'. So really, same as a year and a half ago.

I would point out though that 6 of the people I follow don't have bios, and about 7 just have lyrics.

Data here.


Who Tweets the Most?

These rates are worked out as (total tweets posted)/(total days online). Obviously, the actually post rate will vary over different time scales..

Bubble chart (made with ManyEyes) - bubbles sized by tweet rate (the numbers on some of the bubbles).

The graph below gives a better idea of relative rates, and 'rankings' (click to embiggen)

The blue line is actual values.

The orange is a logarithmic trend-line. It's a pretty good fit (R2=0.95); and, loosely speaking, it means ~70% of the tweets in my timeline come from ~30% of the people I follow. [cf: Pareto Principle]

You get similar log-shaped graphs when you split up the genders.

Full data here.


Chattiest Gender?

You can read all the explanation, caveats, etc. in the previous posts (here and here). I'm just going to go straight into the data.

I follow 27 men and 21 women (excluding celebrities, etc.). The stats are as follow:

Men:
Average = 6.21 tweets/day
Standard Deviation = 6.47

Women:
Average = 13.79 tweets/day
Standard Deviation = 14.27

For clarity, here's a  boxplot (made in R)

Basically, the women tweet more on average, and their rates are more spread out than for the men. In fact, roughly three quarters of the men tweet less than half of the women. Also, there's one outlier in the female group.

This is similar to what we found last time; although the women's average and spread aren't quite as high (average: 13.79 vs 19.21), and the men's average has increased slightly (6.21 vs 5.29).

If you take the ratio of the averages, the women tweet 2.15 times as much as the men. But maybe I just follow particularly chatty women..

Here's treemap (ManyEyes), which should give you a better idea of the gender balance (boxes sized by tweet rate)

Specifically, the graphic above is 62.5% purple (female).

Data here.


Where in the World Are My Followers?

The site I used last time doesn't seem to exist anymore. So I'm using MapMyFollowers instead. As the name suggests, these are my followers, rather than just the people I follow. Nonetheless..

Mostly in the UK and the US. As you'd probably expect.

I will point out though, some of the locations are a little suspect. Some people haven't made their location available so aren't included, and others seem to be in countries they couldn't possibly be in. But it's the best we can do.

Here's a zoom in on the UK

What Do I Tweet?

Made with Wordle, with data from TweetStats.

Words are sized by how often I tweet them; and by extension, @usernames are sized by how often I tweet those people.

In fact, here are the people I 'mention' the most (TweetStats)

Couldn't get a good source on who @replies me. That was one of the things Twoolr used to do..


When Do I Tweet?

Twoolr used to be awesome for Twitter statistics. But sadly, when they left beta, they started charging. And their free service went to shit. Luckily, I found TweetStats. Weirdly, it doesn't need you to log-in or anything, but somehow it can pull data on (nearly) all your tweets - beyond the 3,200 limit. Strange.

Here's some more graphs

Basically, I tweet most on a Friday and Saturday, and at around 1-2pm.

And I've never tweeted at 5am. But that's probably because I'm always asleep at 5am

Except that one time I got really drunk. (SleepBot)


How Much Do I Tweet?

This is another one I used to go to Twoolr for. And, to be fair, I still could. But that only goes as far back as April '10, and its graphics aren't as clear. Here's TweetStats again

Like I said before, I didn't tweet much in my first year. In fact, I only posted 36 tweets in all of 2009.

Now, the one problem with TweetStats is that 5 month gap in 2010. Why is this significant? Well, I was definitely tweeting during that time. In fact, by my estimates, over those 5 months I posted 5,724 tweets (~37tweets/day). So those 5 months account for 43% of all my tweets.

See, the thing is, in 2010, I was out of university, single, and unemployed. I posted a total 8,823 tweets - 24tweets/day. Since I've been back at university, that number's dropped to 11tweets/day.

That lull in Summer 2011 was when I was spending all my time on Tumblr and watching classic Doctor Who. Incidentally, I haven't posted on Tumblr since the start of September '11. It's terribly addictive, you see. I wouldn't recommend it; unless you're addicted to Doctor Who and Sherlock, and have lots of time on your hands..


So yeah.


Oatzy.


[Self-indulgent statistics, and pretty illustrations.]

Social Posting - Part Two: We Post Together

Where were we..

So the main limit to the stripped down model in the previous post is that it only puts people into one of two states - posting, or not posting.

A more realistic model would take into account how much people are posting. The nice thing is, the change to the model is only minor - T no longer has the condition that makes its value has to be either one or zero.

The values T_i now represents the post rate of each user - that is, the number of posts posted during some arbitrary time period, n.

Here is the full, modified equation for the model

Two More Things

As you've probably noticed, we've introduced two new variables. They are:

2) Beta

Beta is a column matrix, with values represent each user's baseline post rate - that is, the rate at which a user would post if there was no-one else online - no audience, no-one to talk to.


1) Alpha

Alpha is a little more complicated. But loosely speaking, its values relate to the upper limit of each user's post rate (ignoring external effects).

It works like this: first, imagine everyone has some maximum post rate. Tbar (T with a line over the top) is the column matrix of these values. Now, suppose everyone is posting at their maximum rate. Alpha values are calculated by rearranging the model equation, thus

Any negative values in W are set to 0, since (in theory) a user has their highest post rate when any of their antagonist are not online.

What this means, is that post rates are capped by their appropriate upper limits; when all members of the population reach their maximum post rates, we find that T(n) = T(n-1) for all n - i.e. the post rates all remain constant, unless externally affected.

For technical reasons alpha is a diagonal matrix.

Alpha and Beta look like this

They won't all ways be size 4, of course. Their size will depend on the size of each particular network. The values of alpha and beta could vary over time, but for all intents and purposes, it will suffice to hold them constant.


Rainbow Graphs

Going back to the graph from last time, we might get a system equation like this

In this full model, it's not just a matter of online or offline. So instead of a series of network graphs, we represent the progression of a network as a set of line graphs.

What we find then is that the system described by this equation seems to tend towards some constant, stable point - regardless of initial conditions.

For example, with the equation described above, and starting at T(0) = [1,0,0,0] we get

And even if we start from T(0) = [100, 100, 100, 100], we get

In each case, the system converges to [15, 10, 5, 10] - Tbar - the maximum rates set when defining the variables for this system.


Interestingly, here's what happens when you set beta = 0 for all users - alpha values recalculated as [1.5, 1.11, 0.5, 1.11]

You get this weird bouncing about. It does seem like they're converging towards some fixed points. And it's converging to points roughly in proportion to Tbar, though much lower.

In fact, in online/offline form, the system looks like this

It's this early jumping on and off -line that causes the system to oscillate at first, before starting to settle down as time goes on.

We also see that, because the system starts much lower than its defined stationary point (Tbar), and because it doesn't have the boost of the base-rates, the actual stationary point the system is converging to is much lower.

If, however, T(0) were set to Tbar, then the system would have stayed constant at Tbar. Similarly, if T(0) were set to [0,0,0,0], then the system would stay there, since it wouldn't have the base-rates to get any of them off the ground.


Antagonistic Altercations

If we set up an antagonist system like that in the last post - with beta and Tbar as above, and conjugate alpha - we get a line graph that looks like this

In this case, the base-rates act to temper A's dislike of B, so that B stays online; though the result is that the stable point is lower than the maximum for everyone. The stable point for A is lowered the most.

But if we remove the base-rates (and recalculate alpha accordingly), the system progresses much like it did last time

That is, as soon as B shows up, A leaves - eventually followed by everyone else.


Upping The Tension

Now, what if we change the setup so that A and B both want to avoid each other.

One might question why two people who dislike each other would be 'friends' (in the social networking sense). I dunno. This is a hypothetical situation meant to demonstrate a point, god damn it!

Here's the system equation

Matrix of maximums, Tbar, stays the same as above at [15,10,5,10].

So if we set T(0) = [0, 10, 5, 10], here's what happens

A comes online because of their base-rate, and once they do, B disappears. Here's this system's initial and stable points

In antagonistic systems such as this and the one before, Tbar isn't a stable point.

In this case, for example, the peak rate for A occurs when B is offline and C is at their peak rate. But C is at their peak rate when both A and B are online. So if B is offline, C can't be at their peak rate, and as a result neither can A. And vice versa. So Tbar can't be achieved, since the tension between A and B pushes down the rates of everyone, themselves included.

The system does still stabilise however. It just to stabilises with B offline, and everyone else at a slightly lower rate in this case.


Now, if we do as we did in the previous blog, and take A offline (indefinitely), here's how the system evolves

In this case, the stable rates of C and D are lower than they were when B was 'forced' offline by A. And even though A isn't online, B isn't posting at their maximum rate.


Exogenous Effects

So as before, we also have to consider external effects.

People can suddenly, and unexpectedly disappear - maybe fall ill, get a job, or just not post for a while. Conversely they could find themselves in a situation where they're posting more - maybe because of a major news event, or because they're at work (procrastinating).

We can quite easily create these effects artificially, then observe how the model responds to that change. For example, we could gauge a person's importance (to a network) by removing them and seeing how dramatic an effect that has.

So in the antagonist examples above, we could say that A is more important to that network than B, since removing B had a greater (negative) effect on the C and D.


For this, we define some arbitrary function, E(n). We then multiply our model equation by this function

So for the example last time, where we removed two users C and D, then later reintroduced C, the function(s), E(n), might look like this

Or, if some major news story broke between n=2 and 5, then the function might look like this

So, if we were to, say, remove A (indefinitely) from the top system above, then the result would look like this

The important thing to note here is that even though B is the only person directly linked to A in the network, removing A from the graph lowers the rates of everyone.

Alternatively, we could double A

In this case, multiplying A by two brings everyone else's stable points up. It's also worth noting that A' stable point isn't just doubled, but multiplied by 2.7. This is a good example of the feedback effect mentioned at the start of the previous blog - increasing A' rate increases everyone else's rates, increases A' rate.

Or if we half A

In this case, everyone else' rates drop too, and A's rate stabilises at less than half it's maximum.


Agitated Antagonist

So what if we go back to our antagonist system, and start poking it...

First of all, if we half A

Then B is able to stay online, but with a stable point much lower than their maximum.

But, here's where things get interesting - if instead we double B, here's what happens

This time, they each scare each other off. But, once the other goes offline, they each come back in the next turn, only to scare each other off again. And so on. And all this jumping between extremes makes C and D jump about as well, but half a cycle out of sync.

At least, that's what happens for initial conditions [0, 10, 5, 10]. If we start with all at 0, the behaviour is nearer to that with no external effects.


In General

So it seems to be the case that externally changing one user affects (almost) everyone in the graph.

That being said, I would expect that when the model is generalised to much larger networks, we would find that the greater the distance between two users, the less affect they will have on each other.

It probably stands to reason then that, for a given user, it's only worth considering a graph as wide as friends of friends. Maybe friends of friends of friends for (a collection of) dramatic changes at that distance.

More in general, when you apply the model to a system of more than four people (as we've looked at here), you would expect things to get more complicated.

But the overall results should be similar - that is, in most cases the system will tend towards some stable state. It may just take longer to reach that point, depending on the size of the graph.


In Conclusion


So the long and short of it is this - in most cases, a group's post rates will typically converge towards some stable point.

If there aren't any antagonists, and if the base-rates aren't zero for everyone in the network, then the stable state will typically be the common maximum, Tbar. Otherwise, the system may oscillate, or else become stable at a much lower set of rates.

If we set all base-rates to 0, we find that the systems behave much like they did in the simplified online/offline model discussed in the previous blog; with some quirks of their own.

In the absence of base-rates, or in the presence of antagonists, initial conditions become important, affecting if and where the system stabilises.

When we consider external effects, the same will tend to be true - but with the stable points altered, according to what that effect is.

Modifying one user affects everyone else in the graph (to some degree). Modifying different users will have different effects on the rest of the graph, depending on the 'importance' of that user.

Of course, in the real world, external effects are unpredictable, and will tend to keep the system off-balance (unstable). But we do what we can.


Oh, and if you were wondering how I got the numbers for the graphs, here's the code I used. But be forewarned, it's very sloppy.


Oatzy.


[..and the fundamental interconnectedness of all users.]

Social Posting - Part One: Who's Online?

Coherence of Absence

So imagine it's Sunday, there's nothing to do, nothing's open, nowhere to go, nothing on TV. So why aren't people spending all this free time online - tweeting, Facebooking, whatever?

Sunday is an online dead-zone.

A random hypothesis I thought up to explain this, posits that this may be the result of large numbers of people thinking "why post, if there's going to be no-one around to see it?". After all, no-one's around on Sunday. If they were, they'd be posting stuff.

I called it Coherence of Absence, because a fancy name can make a daft idea sound respectable. The alternative was The Sunday Effect.

Of course, it could be that people mostly social network when they're at work (procrastinating). Or maybe they're asleep/hungover on Sunday. Or have nothing to talk about, since Sundays are so boring. I dunno. I'm unemployed, so Sundays aren't much different to any other day of the week.


In Other Words

So this can be boiled down to two principles:

1. People are likely to post more when there are more people online

2. The number of people perceived to be online at any given time is based on who is posting

It's fairly intuitive that when there are more people online (posting), you'll post more - not just because there are more people to talk to - tweets to reply to, posts to comment on - but also because there's a bigger audience; more people to see what you have to say, regardless of how interesting or mundane that might be.

In essence, these two together create a feedback effect where your posting habits are based on your friends' posting habits, are based on (to some extent) your posting habits. And so on.

For point two, we can't generally say for sure whether or not someone is online, except for when they post something. Though there are exceptions. Of course, the irony of this model is that it assumes that everyone is online all the time - just waiting for someone else to indicate their presence with a post.

Don't get me wrong, I'm not saying this is definitely how the world works. This is more of a, let's suppose this is how it works - what then?

NB/ the people a user follows are referred to as 'friends', since 'the people a user follows' is such a mouth full. These imaginary people may not, strictly speaking, consider each other friends.


Technically

Right, let's dive in: for some network of users, define a matrix M with entries

For example

NB/ connections between users needn't necessarily be two-way (except in the case of Facebook).

Next, we define a column matrix, T(n), with entries

Now, we say that if at least one of a user's friends is posting, then the user will 'come online' and post in the next turn. Vice versa, if none of their friends are posting, then the user will 'go offline' and not post in the next turn.

To do this mathematically, we just multiply the matrix M by the matrix T. For example, if we start with only B online (blue),

Then, as expected, all the people who follow B come online. B goes offline since none of the people they follow were online in the previous turn.

You'd then multiply M by T(1) to get T(2), M by T(2) to get T(3), and so on. Here's another example, starting with only A online,

Easy.


That's Not Normal

The problem with the model so far is that there only needs to be one other person online for a user to join them. But one person online will be more significant to a person following 5 people, than to a person following 100.

So we might instead say, if at least half of a user' friends are online, then the user will join them in the next turn. First of all, we define a matrix, W, with entries,

So for the example above, we get

That is, the matrix M normalised so that the sum along each row equals 1. Our definition of T then becomes

Which is just a fancy way of saying, if at least half of a user' friends are posting, then the user will post in the next term.

So, for example, if we start with A and D online,

As before, everyone ends up online. But in this case, if we had started with only A online, then that means only a third of B' friends are online, so B wouldn't come online in the next turn. Nor would anyone else, for that matter - everyone ends up offline.

Of course, you could change it so that a user will come online if a third, or a fifth, or whatever of their friends are online. If you really wanted to.


Weightlessness

Of course, this supposes that a user cares about all their friends equally. In reality, we might care more about one person being around than another. There may even be people we don't care about at all (in terms of what we post), or people that we want to avoid.

The trick is this - instead of assigning everyone a 1 in the matrix M, let user i assign to each of their friends, j, some numerical weighting, based on how much i cares about j. If i prefers to avoid j, then this value may be negative.

Call this matrix W'. This matrix is, again, normalised to generate W, as before; making the sum along each row equal 1. The only difference is that with antagonists, some rows may add up to 0. So to avoid division by zero we have this

At this point, I'll introduce the equation for the model so far

Avoid the Antagonist

So let's say this is our network now, with accompanying normalised-weighting matrix

B has no qualms with A, but for whatever reason A harbors some secret dislike of B, and will typically want to avoid them. Why does A still follow B? Keeping up appearances, stalking? I dunno. Let's just pretend there's a good reason.

So, for example, if we start with A, C, and D online, here's what happens

B comes online because both their 'friends are online. But B coming online effectively scares everyone else off, one by one. Admittedly, that's partly because I specifically chose weightings that would cause this to happen. But you get the idea.

Of course, once B goes offline, A might decide to come back online again, with D and then C following shortly after. At which point, we end up back at the start (1), where the cycle repeats.


Can't Talk

For this we go back to the first graph above, weightings all the same. Say one or more of the users go offline - maybe they're out on the town, or asleep, or something of that sort. What then?

If we take just one person offline, then nothing will happen to any of the other users - except if you remove B, in which case only A goes offline. So in the below example we take C and D offline (red)

By (4) everyone is offline, so that when D comes back online at (5) there's no-one else online to keep them around. And D alone isn't sufficient to bring anyone else back online.


Wait For It

So this is a very simplified model. For one thing, it only considers people as either online or offline - or if you prefer, posting or not posting. It also assumes that, in the absence of an 'audience', a person won't post. And that leads to the implication that a social network can be destroyed by strategically removing users. It can't. I don't think.

Anyway, til next time..


Oatzy.

Tumbling, Part Two: Over the Threshold

I Didn't Vote For This

A short while back, while people still cared about AV versus FPTP voting, I wanted to come up with a voter simulation, so as to get data to compare the merits of the two (and a few other) systems, since real-world data is limited. Sadly, it came to no avail, since I had trouble coming up with a plausible way to simulate the behaviour of British voters.

The basis of how it worked was this - each voter has ten 'vote points'. These points are divided amongst the parties according to how much the voter likes each party.

So if a voter was a strong Conservative supporter, they might give all their points to Conservative. A voter who's more liberal might give 6 points to the Lib Dems and 4 to Labour. Or whatever.

In the case of the simulation, these points were going to be assigned randomly, but according to some Markov Chain type model - based on the typical voting habits of Brits. This, by the way, is where I was having trouble - for large numbers of voters, the result were always the same.

The other part of the model was to (randomly) assign to each voter an 'approval threshold' - that is, if a voter assigns to a party a number of points higher than their personal approval threshold, then they 'approve' of that party, and will place a vote accordingly.

So if, in the liberal example described above (voter B in the diagram), the voter has a threshold of 5, they would place a vote for only the Lib Dems; if, on the other hand, they had a threshold of 3, then (under an AV voting system) they would place a vote for both - order of preference: 1. Lib Dem, 2. Labour.

(In the diagram, all three 'voters' are given the same approval threshold for simplicity. The thresholds are by no means constant and universal.)


I Like It, But Not that Much

So what does this have to do with Tumblr? Well. On Tumblr, you can either ignore a post, like it, or reblog it.

Now, there are some people who will reblog almost anything that catches their fancy. Then there are people, like me, who are very picky and only reblog things they really like.

This is basically analogous to the approval threshold in the voting sim.

Obviously, the quality of a post is subjective (to some extent). But imagine first that for every post you see, you assign to it some rating from 0 to 10. The rating you assign to a given post will be (relatively) unique to you.

Here is how we get around the problem of subjectivity,

We assume that the ratings a large sample of people will assign to a given post will have some regular distribution - most likely a normal distribution - where the average and spread of ratings reflects (to some extent) the quality of that post. Or, at least according to that particular sample of people.

Each user then has two threshold values - a 'like threshold' and a 'reblog threshold'. And as before, if the rating a person assigns to a given post exceeds one of these thresholds, that person will either like or reblog the post (possibly both) accordingly.


Some Asides

It's worth noting that similar modelling can be applied to any piece of information that can be spread - whether it be on Twitter, by email, whatever - where you have some 'sharing threshold' by which you judge whether or not you think a thing is worth sharing.

This may vary wildly from situation to situation. But in most cases outside of Tumblr, you will tend to find that people typically have a higher share threshold - higher standards, are more pick, are more wary of what they share, etc.

Another thing to consider, with Twitter in particular, the originator of a post can have a massive effect on how much said post is retweeted.

For example, some people (fangirls) will retweet someone like Justin Bieber simply by virtue of the fact that it's Justin Bieber. Conversely, some people may not retweet someone like Piers Morgan because they don't like him - even if they would otherwise like something he posted enough to have retweeted it.

Finally, people don't always share things simply because they like them - they may want to make some comment on the post, they may want to mock the post, whatever.

But for simplicity, we're ignoring that all these eccentricities, on the assumption that including them would have no appreciable effect on the model.


Model Affinity

I wrote a quick model of this behaviour (Python, can be viewed/downloaded here).

For this model, the user's thresholds are generated uniformly - that is, all numbers in the ranges have the same probability of being chosen,

For the 'like threshold' you pick a random number between 1 and 11.
For the 'reblog threshold' you pick a random number between the 'like threshold' and 11.

The user's assessment of a post is randomly generated based on a normal distribution - in this case with mean=5, sd=3, constrained to the interval [0,10]. These values were chosen arbitrarily.

For a simulation of 100,000 users, this model returns approximately 25,575 likes and 14,487 reblogs. This gives a 'reblog to like' ratio of 0.56.

This is quite different from the 'R:L ratio' of the Dr Who image, discussed in the previous post - which turns out to be 1.42

But if we (effectively) increase the quality of the post - mean=10, sd=2, same interval - we now get approximately 32,393 likes and 41,872 reblogs, giving a R:L ratio of 1.29

Which reflects reality quite well - that is, the better a post is, the more reblogs it will get (and the higher its R:L ratio). You could also get a similar effect by lowering the approval thresholds of the users - effectively lowering the users' standards.

Note/ You could probably get a better replication of real world data by fine tuning the random variables. The ones used here were chosen fairly arbitrarily.


Personal or Probable

For large numbers of people, there are two ways you could work this into a model of post spreading:

1) On a per person basis

That is, for each user work out whether they will ignore, like, or reblog, as you go. This would lead to richer behaviour on a per-simulation basis, and is better if you want a more detailed, more 'precise' model.

2) Based on the averaged probability

That is, run a simulation for a large number of people (as above), work out the average number of likes/reblogs, and use this as the 'probability of liking/reblogging', constant for all users. So from the first simulation above, the probabilities would be:

p(reblog) ~ 0.14
p(like) ~ 0.26

This would make the behaviour more uniform across multiple simulations, but would be less computationally taxing.

It can be argued that, when you average over a large number of simulations, you will get (almost) identical results for both approaches. So it's ultimately a matter of preference.


In Part 3 of the series, I'll be looking at constructing a model of how posts spread through Tumblr, building on the model described here.


Oatzy.


[addendum] -  Due to a mild error, the section labeled 'Model Affinity' had to be rewritten. If you read this post before this addendum was added, it's probably worth re-reading that section.