New FB Comment System Analytics: Comparing TechCrunch.com Before and After The Switch

Alistair Croll & Sean Power blog on watchingwebsites.com about web technology, startups and analytics.  They are the co-authors of Complete Web Monitoring (O’Reilly, 2009) and contributors to Web Operations (O’Reilly, 2010).  If you like this post, you’ll probably like the one we wrote about analytics &  the TechCrunch bump.

Facebook recently introduced embedded commenting within websites, under the name Facebook Comments. Does this new model for commenting on posts help, or hurt, site engagement?  To find out, we compared two weeks’ worth of TechCrunch posts; 7 days before and 7 days after the site implemented the Facebook Comments feature.  Data geeks, you can find the source data here.

In order to reduce the outlier effect of posts with very high, or very small, levels of comment —the massively successful, or truly abysmal, ones—we trimmed the lowest and highest 5 percent of results. In this analysis, “All posts” refers to sums and averages on all posts published, whereas  “average posts” refers to sums and averages on all posts with the 95th and 5th percentile values removed.  By trimming the results in this way, we hope to get a better representation of the effect of Facebook Comments on a typical post.

TL;DR

  • For all posts, implementing FB Comments caused a 42% reduction in the total amount of comments, and a 38% reduction in comments per post.
  • For the average post, implementing FB Comments cause a 58% reduction in the total amount of comments and a 56% reduction in the average amount of comments per post.

In other words, TechCrunch saw almost 50% less comments when they implemented Facebook comments.

But it’s not all doom & gloom.

People liked content more often.  This probably led to a greater number of incoming visits from FB.com, but I don’t have the analytics to prove it.  Both Erick and MG have stated that FB referrals have skyrocketed.

  • For all posts, implementing FB comments caused a 27% increase in the total amount of likes, and a 36% increase in likes per post.
  • For the average post, implementing FB comments caused a 14% increase in the total amount of likes, and a 16% increase in likes per post.

In all cases (with and without outliers), “google buzzing” increased by 30% in both the total amount of buzzes and the amount of buzzes per post.

However, it’s notable to see the impact this had on Tweets

  • For all posts, implementing FB comments cause a 4% decrease in the total amount of retweets, and a 2% increase in the amount of retweets per post.
  • For the average post, implementing FB comments cause a 1% decrease in the total amount of retweets, and a 7% decrease in the amount of retweets per post.

What’s Missing

To be able to fully understand the scope of Facebook Comments, we’re missing a few critical factors which are only available to the owner of the site itself:

  • Visits / post
  • Referrers / post
  • Revenue / post
  • Time Spent on Site / Referrer
  • New vs Returning Visitors / Post

In other words, it’s important to measure the amount of interest visitors showed by the channels that brought them there.

What this means if you’re …

… A High Volume Media Site / Blog

If spam or trolling is a big problem for you (it probably is), the Facebook Comments platform is a viable method to solve this issue.  You may encounter backlash from the community.  Expect numbers to initially dip before stabilizing.  Make sure you track the above numbers diligently, and give yourself at least 2 weeks (preferably 4) to fully understand what you gained and lost.

A Medium to Low Volume Media Site / Blog

Chances are that you’re still in reader acquisition mode.  Facebook commenting is not a viable solution as it stands today, until it implements the ability to authenticate via other platforms (yahoo, twitter, etc).  Consider implementing if you’re having issues related to abuse, trolling or spam where anonymity is not a requirement.  Otherwise, stick with Echo, Disqus, etc.

Facebook

Your strong value proposition comes in two forms: your ability to drive users from your own platform to publishers, and your ability to prevent spam and trolling by forcing identity on all comments.  If you can claim the largest publishers, you have a chance at usurping Twitter’s position as the leading means of spreading awareness about a piece of news.

A FB Comments Competitor (Echo, Disqus, Etc)

With Facebook entering your market (and Google not far behind), you need to concentrate on providing excellent user experience for your commenter.  Your greatest asset is a community of users demanding that your system be kept / implemented over those of your competitors.  Consider creating ACLs that allow publishers to force users to authenticate via certain ways if you don’t already have them in place.

Twitter

With the implementation of comments, Facebook has the chance to significantly increase their ability to socially propagate publisher content, and consequently, their stake in the social media landscape.

  • Anonymous

    I wonder about the long term effects of using FB Comments. I personally wouldn’t want to comment on a site that uses this service since I don’t want to spam my FB friends’ news feeds with out of context comments I make around the web. I know there’s a checkbox to disable this default behaviour, but I figure I’ll miss it from time to time (does it save your choice?).

    So if other users are like me (and judging by the stats there are quite a few), how will this affect long term engagement? Maybe after a few weeks I’d get fed up with a site that uses a comment system like this and switch to another one. I guess more time is needed to tell, and then the info may not be available to the public, oh well.

    • http://twitter.com/seanpower Sean Power

      yeah, i really think there’s a baking period that we haven’t yet reached. Over time, I get the feeling that we’ll see people conforming to this behavior much like they did with the news feed changes and so on. But again – time will tell.

  • Anonymous

    I find the comments now to be blander and not as informative, the real skinny was from people posting anonymously. Any thoughts on quality?

    • http://twitter.com/seanpower Sean Power

      We purposefully tried to shy away from subjective ranking of comments. Besides, TC, Scoble and others have done a great job of covering that side of FB comments. It’s hard for me to make a blanket “yes, comments are better / worse” statement without having the data to back it up. I get the feeling that FB comments will eliminate outlying comments in favor of more consistent conversations. In other words – less 4chan, more Quora. This statement is not based on data and simply are simply gut feeling-driven.

      • Anonymous

        Reducing outliers is a good way of thinking about it, but the implication is that to get rid of the dross, you get rid of the nuggets too – what is at issue is if those nuggets are more valuable than blandness (I assume its a volume vs CPM trade-off).

        • http://twitter.com/seanpower Sean Power

          Indeed. that’s why we gave numbers in both cases – with and without outliers. It’s also important to note that this is for a week’s worth of data. To get truly meaningful numbers, it would be important to trend this over a few weeks (before and after).

      • http://www.bizsparked.com Ed C

        Hey Sean, thanks for coming up with some data for us to play with.

        I understand why you’d want to “shy away from subjective” in most situations, but in this particular case, quality of comments is just about the only thing that matters (even if it’s harder to generate numbers for). Reading through comments is like digging for gold. If we just measure how much dirt there is to move, it’s really missing the mark.

        Maybe there are some proxies for quality you can use (e.g. comments with more than a couple likes)? If the pattern of “liking” has changed pre/post FB, you might also need to normalize the two scenarios separately.

  • Anonymous

    MmmmMMmmm data, yum yum yum

    • http://twitter.com/seanpower Sean Power

      nom nom :)

  • http://www.scanman.com.au/ ground penetrating radar

    Boom boom

  • http://rynoweb.com Chuck Reynolds

    My opinion is it was a good move for techcrunch & those big guys – killed most of the trolls and brought some more authenticity and transparency into the comment threads.
    Smaller sites… it’s a struggle to get anybody to comment. Each site is different though and I think the “baking time” for FB comments is a ways off. They do need to implement twitter but do you honestly see that happening? FB’s not a big fan of Twitter last time I checked lol.. Same with Google – but that’s all politics and handshakes that can fix that.

    • Andrew

      they integrated yahoo interestingly

  • http://www.twitter.com/bartificial Bart

    You don’t think that a switch to any new commenting system would cause a drop in comments? So if Facebook Comments was the standard on TechCrunch for a year and they switched to something else (even their old system), they would see a drop then too? If I were doing this test I would probably take out the 7 days following launch, not using it as the primary example.

  • Anonymous

    I’ve stopped commenting on TC entirely and my visits are less frequent. I do not want my web persona and my personal FB account linked in any capacity. And I personally feel the comment quality is way down. That said, I’m sure the tradeoff in users like myself becoming significantly less engaged versus the elimination of trolls combined with a spike in traffic is worth the loss of my most excellent comment insights and eyeballs. It would be nice if there were a way to create an anonymous commenting profile on FB that shielded my persona on comments, but still let TC know my FB as a means to kill trolls.

    • http://natts.com Dave Nattriss

      Your web persona and your personal Facebook account are separate? How strange! Why do you need to post anonymously?!

      • Erythros

        Well, it is not strange. I would say that it is pretty normal or at least insightful. First off, he is not posting anonymously! He is posting under his identity. And I bet that he posts under this identity on many pages and blogs and e-zines and so on. You see, he is building his identity based on his comments and opinions. That is not anonymous, that is the direct opposite.

        So what if his web and commenting identity is different from his FB identity? Everybody does that. Using only one identity is very stupid. One does not go to kindergarten to pick up his children in body harness in which he would go to local BDSM party.

        • http://natts.com Dave Nattriss

          If the identity is anonymous (not tied to his real identity), what does he actually achieve from building it up?! It’s completely anonymous because it doesn’t tie up to a real life person (with a real job and real family and real friends and real possessions etc.).

          If you understood anything about Facebook, you’d know that you can restrict different parts of your profile and posts to the different people that you know. You only need one identity, but with filters, which is what Facebook provides.

  • http://twitter.com/mauricioswg/ MauricioSWG

    I wonder if this model will remain at TechCrunch because it attracts referrals from Facebook “Likes.”

  • http://paulgailey.com Paul Gailey

    the @financialpost also are doing a trial of running FB comments, which is arguably more interesting than the Techcrunch experiment. Perhaps somewhat extreme as Financial readers maybe more resistant to using their FB profiles to comment, but if FB comment experiments widens the commentsphere then surely it merits serious consideration by publishers?

  • http://groupon.comclone.com Groupon Clone Php

    hmmmmmmm hmmmmmm.

  • http://techkik.com Amrita Mathur

    Wicked analysis, thanks for sharing Sean!

    I especially love how you have made the disticnting beween high volume sites and lower volume. In a nutshell, using Facebook Comments exclusively, does not solve the problem of anonymity.

    (My blog post from a while back: http://techkik.com/2011/using-facebook-comments-for-your-business-or-blog/#comment-228)

  • http://blog.ianlyons.com Ian Lyons

    Thank you for not making yet another simplistic comparison of large numbers being better than small numbers. I’d wager that nobody (sane) reads through hundreds of comments – particularly if many of them are low quality. Any system that creates more salience is a step in the right direction. I’m no FB fan boy and a $65B software company should be delivering much better technology but it’s still moving in the right direction.

    My personal experience of Techcrunch moving to FB comments was that a comment from a friend (whose opinion I respect) was surfaced near the top, triggering me to comment and over the next few days I was drawn back to the article because of reply notifications in FB. I have never before commented on TC – so the new system caused me to engage. If you’re just counting numbers, you’ll love trolls – if it’s community and engagement you’re after you’ll have to take a serious look at this.

    • Leland

      Another blind Facebook follower thinking that using real life ID for everything is appropriate and encourages better discussion in an environment designed for anonymity. Sigh.

  • Blearning

    I agree that sometimes there is no real advantage to adding more features such as intertwining all the SM experiences together.

    Bill learning
    http://myonering.com

  • http://Twitter.com/Jacob_Walsh Jake Walsh

    It’s interesting that you decided to remove outliers and focus more on the effects it had on the “average post.” It makes sense at first glance, but I think it might be a little short-sighted. Here’s why:

    One of the biggest advantages of using Facebook Comments is that you have the potential to reach thousands of people who normally wouldn’t read your blog. This means the “massively successful” posts (as you called them) have the potential to be exponentially greater than they were when the comments were isolated on TechCrunch. The more comments you pump into the Facebook News Feed, the more exposure you get to new readers. That means over time the massively successful posts will draw in more regular readers, and as a result they should start to see comments climb back up for “average posts.”

    In other words, the positive outliers are going to play a major role in increasing comments over time. It’s just going to take more than 7 days for that to happen.

  • http://ww.iControl.tv Shakir Razak

    Hi,

    Ever since they implemented the new system, I stopped both commenting and reading the comments – they look ugly, and knowing that if I read, I’d want to react, but not be able to, there’s no motivation, so I just read the articles themselves more superficially when I do.

    Previously, I would read comments in interesting insightful posts/conversations even beyond a hundred deep.

    The amazing thing I find is that anyone who was tech.savvy and aware of the potential of Facebooks’ data-black-hole and value-ownership, would know not to commit more then what is necessary at the least, and not even sign-up to it in the first place at the most concerned, and I am commited as far as possible to not to do so.

    That means that any business that pins itself to the borg will lose me and any other consumers worried about the potential, or simply on other platforms, to the detriment of their own business and readership, they are passing on the value to a 3rd-party and losing out when those locked-out might have brought new custom or content/insight.

    I also, never commented annonymously, always, even when arguing/dismissing, putting my name to my online words, but there is no chance I’m going to let a company like facebook be my digital avatar/id and retain all that would be within.

    I hope Techcrunch sees sense, and goes back to an open platform, as it is, I can’t even argue my point to be heard.

    One analogy would be compare it to the difference between a park and a private club/party, both are enclosed spaces of sorts, but one allows anyone to come in and take part, and the other one has tickets and bouncers at the entrance – would the internet and web 2.0+ be where we are if those had been the pre-conditions in the first place.

    So reminds me of the old battles of Web1.0:

    http://query.nytimes.com/gst/fullpage.html?res=990CE3D61F3CF932A05754C0A963958260&pagewanted=all

    Yours kindly,

    Shakir Razak

  • Anonymous

    Good analytical write up on embedded comments by FB..would like to inquire if I could link back to your site from my website
    (Which is being launched)  www.tbbos.com which discusses Ways to make
    money online
    , How to start
    online business
    and has a Directory
    of online
    business
    . Tbbos also offers free
    ads
    for members. If you are interested, please mail me on admin@tbbos.com

  • Anonymous

    I don’t like that of comment. i have realize the article or press release and think the comment post.

    http://www.techzar-web-developers.com/

  • Cy00800

    HOGAN presa hogan hogan australiani o no hogan o not.Men rapida Hogan, foxy brevi, lo esterni sono eccellenti per rhoganed terrain.It ‘s difficile telefonare questi “hogan” – retribuzione durante il vero senso della parola,outlet hogan sono not.Although agevolando programma, questi negozi ti aiuta ad accedere a un’ampia gamma di collezioni a rates.Nonetheless equo, con la prosperità dell’impresa internet business, Hogan non è così 
    costoso come anticipo, si può scoprire a prezzi accessibili in vendita Hogan scarpe hogan i due nella rivenditore e online, con tutta la recrudescenza di ottenere a basso costo Hogan, sta andando a ulteriore sconto in inverno season.Thirdly, cercate di ricordare di mettere i calzini di cotone quando si effettua un tentativo di Hogan per la ragione hogan interactive che Hogan sono essenziali 
    hogan in inverno, poi calze di lana e collant sono il vostro partner molto meglio quando indossano ornamenti hogan.Removing incredibili,hogan uomo hogan pelle di pecora australiana persone presenti tutti options.Not aggiuntiva ha un super-modello alto figure.Brad Pitt, Justin Timberlake, Leonardo Di Caprio e molti attori sono normalmente hogan online molto più notato in Hogan in film 
    unici e celebrations.But come può essere l’umile Hogan secondo created.In con il produttore 
    Hogan, Hogan Cardy è eseguire una dimensione totale larger.hogan olympia six.promptly in seguito si 
    colpisce la strada, si femmine benefici autorità, maschi, giovani più piccoli allineati 
    neonato hogan disattenti outlet.So,hogan prezzi ecco alcuni consigli per quanto riguarda il modo migliore per cercare subito dopo le Doc Martens: In primo luogo, di eliminare i lacci (ora sarebbe il momento ideale per ordinare alcune sostituzioni in realtà si dovrebbe mai li richiedono) e pulire la vostra hogan completamente utilizzando un panno pulito o forse un morbido hogan brush.So signore, non essere esitate, Hogan è il guardare per ora presa hogan.They shoganest si ordina una piena dimensione collezione hogan oltre a un mezzo più piccolo se siete tra Hogan sizes.The volta insuperabile per 
    voi di incontrare le hogan di pelle di pecora individuali per l’inverno è particolarmente 
    hogan 2011collezione Hogan prima del 
    starts.These periodo di lana fatta hogan in grado di offrire il comfort e conservare i 
    vostri piedi da day-to-day.Your calzature è responsabile salvare hogan rebel i vostri piedi da questo freddo 
    e l’aria fredda climate.Let hogan secca – non asciugare che ha un asciugacapelli o della 
    domanda sunlight.Their diretta aumento ha prodotto loro preferito a livello mondiale e 
    impostare un ampio mercato per la sua marcata traits.hogan scarpe For quelli con le gambe sottili, che dovrebbero spendere particolare attenzione a prescindere dal fatto che c’è qualche gap che coinvolge le gambe e hogan.Phase uno -.outlet hogan Tanning / Elaborazione step iniziale nella concia delle pelli o la trasformazione di pelle di pecora è “ammollo” le loro varietà di array pantofole hanno dato prezzi Hogan una miscela di placare e gentilezza sul buds.Hogan crescente di bambini piccoli possono essere facilmente accessibile al vostro finish.It è possibile usufruire rapidamente queste magnifiche collezioni dall’alto mercanti di linea, ma poco prima che lo rendono sicuro, si potrebbe essere avvalso con pelle di pecora genuina maiale.