My first article for Co.Labs was about a 65-year-old Android engineer. It hit the number one spot on Y Combinator's wildly popular Hacker News, greatly increasing readership, but I never thought to investigate how it got there. Google developer Ken Shirriff has.
Shirriff analyzed the supposedly meritocratic system of Hacker News rankings and found that the fate of the front page stories is not just determined by votes, but also by a mysterious system of “penalties” which could push stories with certain intrinsic characteristics down the rankings.
Shirriff tracked the top 60 Hacker News stories for several days and found that 20% of front-page stories and 38% of the articles on the second page had been penalized, causing them to drop rapidly down the rankings.
Shirriff’s own blog post on the results of his study became one of the penalty systems’ victims, being forced off the front page after a penalty was applied. Without the penalty, he claims the post would have appeared at the #5 spot.
According to Shiriff the basic Hacker News score is determined by the number of upvotes, the time which has passed since it was submitted, and any penalties applied. Time is weighed more highly than votes to make sure that nothing stays too long on the front page, but penalties also play a big part.
By charting the fate of the top 60 stories stories on November 11, Shiriff showed that the #1 spot (the red line in the graph) was frequently not occupied by the most popular story. A story called "Why You Should Never Use MongoDB" should have spent much of the day at #1 based on the raw score, but it was penalized to the degree that it mostly lingered at around #7.
If an article receives a penalty factor of 0.4, every upvote only counts as 0.3 votes. A penalty factor of 0.1 pushes each vote down to counting as only 0.05 votes. Shiriff’s estimates that his own article suffered a penalty of 0.2. It’s not clear how and why penalties are applied but Shiriff noticed that many penalties were applied at around 9 a.m. and theorizes that this could be when moderators start work.
Articles appear to be commonly penalized for controversy, indicated by attracting more than 40 comments, for coming from a popular website like ArsTechnica, Business Insider or GitHub, or merely for including "NSA" in the title.
Commenters on Hacker News pointed out that the analysis didn’t factor in flagging. If an approved Hacker News reader--flagging is not available to all users--thinks a post is spam or off topic, he can flag it for removal.
[Image: Flickr user With Associates]