Jonathan Levitt has spent the last 5 years as a pioneer in the voice of customer analytics space. Through his speaking, writing, and evangelism, he was instrumental in legitimizing voice of customer analytics at a time when traditional web analytics still dominated the online business intelligence conversation. Jonathan has worked with world leading brands like Bank of America, Verizon, Dell, Procter & Gamble, Ford, and Reebok and has been featured in several industry publications including 1to1 Magazine, ClickZ, DM News, and MediaPost.

One of the best sources of business intelligence for companies of any size is raw Voice of Customer data.

This is particularly true for start-ups, where early, frequent, and consistent interaction with customers is critical to getting off the ground. The more customer-centric your decision making processes are from day one, the more likely you will get to the next stage in the development and maturation of your business plan.

This explains the recent growth in the selection of free and low cost Voice of Customer collection tools. User Voice, Kampyle, Survey.io, 4Q Survey (disclaimer: I helped conceive and build 4Q) — all of these are examples of popular Voice of Customer collection tools that can provide site owners with a pipeline of cheap and actionable visitor-sourced insights.

Once you put on the VoC practitioner’s hat, however, questions about respondent count size inevitably come up. Simply put, you need a way of knowing how much data is enough.

At what point can you act on the findings coming through your shiny new tools, with full confidence that you have collected a representative sample of your audience? If you’ve been running a User Voice customer feedback tool for 3 weeks and you’ve only collected 20 respondents, is that enough to act on? These are certainly agonizing questions for a data-centric marketer.

Now’s the time to start glancing over enviously at the big sites, because they don’t have this problem. The laws of probability are such that feedback from 500 respondents is usually enough to deliver reliable data at even the strictest confidence intervals. A big site like Dell.com can pull in 500 respondents within a day or two; at that clip, statistical significance comes through in a heartbeat.

But since your traffic generation muscle isn’t likely to match Dell.com’s anytime soon, I’ll let you in on a little secret: for small, startup websites that want immediate answers to their questions, the size of your sample almost doesn’t matter.

Here’s why. Representative feedback sampling requires a known population that is relatively stable and doesn’t fluctuate all that much–basically, a predictable population that will yield reproducible results. But the visitor bases of small, startup websites are anything but stable, especially if the websites are in a voracious traffic acquisition mode. The reality is that the composition of their online audiences is constantly shifting, which seriously undermines any effort at scientific VoC measurability.

This is one case where directional data can be just as powerful as representative data. I’m not saying you should blow up your website and start from scratch because of 1 piece of negative feedback, but you don’t need more than 20-25 pieces of feedback to really get started. So, forget about the science and the stats, and focus instead on segments of visitor discontent.

Look for repetitions and common patterns in your feedback; group similar items together and focus on sectional site optimization . If you notice two or three pieces of feedback that are eerily similar, then chances are your respondents are surfacing a real issue that’s resonating far deeper in your growing visitor base.

Voice of customer research can be a wonderfully responsive early warning system for a small website owner. Don’t get caught up in obsessing over respondent counts. If you’ve got 25 or so pieces of real visitor feedback at hand, you can go a long way in constructing a visitor-centric website experience that will help your website to grow and flourish.

Jonathan Levitt

Blunt – The Conversation Agency

There’s been alot of talk lately on the idea of prototyping and demonstrating a product to solicit feedback.  The catch?  The product isn’t actually built.  This goes hand in hand with the MVP concept practiced by lean startups.

I was on BackType today, and I just witnessed an awesome example of this concept in action.

BackType is a search engine that indexes millions of comments across social media platforms and lets you query for particular topics that interest you.  Cool, right?  I performed a search query for “analytics”, and the resulting page had a small tab called “Trends”.

Curious, I clicked on it and was brought to this page:

Drat, it’s not released yet!

Brilliant!  BackType has given me an excuse to come back and check to see if the tab is active.  Even better – they were able to collect my expectations before the feature has even been released (or built!).

This is product management crowdsourcing at its finest.

Well done, team BackType.

Disclaimer 1: All site-related data found in this post comes from compete.com.  The company was kind enough to give us a “pro account” to help us research the O’Reilly book that we wrote called Complete Web Monitoring (thanks, you rock!).  However, compete.com did not sponsor this post (nor did any company, for that matter).  And yes, we know – compete.com numbers are simply estimates.

Disclaimer 2: I (Sean) worked for Akoha as Community Gardener while we launched at TechCrunch50 2008; but I’m now doing metrics, web analytics, performance, and social computing consulting.  The views found below are mine, and do not reflect those of Akoha in any way.  For the record, Akoha is awesome!

About us: This post was written by Sean Power with Alistair Croll.

Dear TechCrunch50 Startups,

Congratulations. You made the list. You’re finally launching, and that pent-up frustration of not being able to tell people about it for a month is almost at an end. Now, you have to live with a weekend of cold, hard fear that your demo will explode. You’ve got an interesting week ahead, and I know you’re short on sleep, so let me get to the point quickly.

You’re probably excited about the TC50 bump. I first saw the term used by Josh Kopelman of First Round Capital on the RedEye VC blog. The bump refers to the pounding your website is about to experience from TC50 attendees, readers, bloggers and their friends.  It’s not to be underestimated.  Here’s a glimpse at how the bump looked like for all TC50 startups in 2008.  If you squint a little, you’ll see Akoha somewhere in there!:

This is an unprecedented influx of attention. It may be the single biggest traffic spike you’ll ever experience. Thousands of visitors will drive by your site, stay for a minute, and leave — never to return. After the bump, you’ll feel a tremendous rush of adrenaline, then deep, soul-sucking disillusionment as your traffic dwindles back to its former levels.

Don’t waste this opportunity. If you take the right steps, you can make the most of your fifteen minutes of fame.

Being Talked About Is Nothing. Being Remembered Is Everything,

You’re probably celebrating the traffic you’ll get to your site. But you shouldn’t be, unless you capture the minds and hearts of your visitors.  The best way to do that is to target them.

On a normal day, you know very little about your site traffic. But today, you know lots. And that means you can tailor the experience to your audience. Let’s say you knew for a fact that 95% of your visitors loved the color red.  You’d probably redesign your site to make it bright red, right?  Well, next week, over 85% of your visitors will all come to you through TechCrunch somehow. Think about it. For a single week, you know exactly what kind of visitor will be visiting you.

TechCrunch50 visitors will load your site, scan quickly, and may even register (if registration isn’t broken, and is simple enough). Then they’ll leave.  In that crucial 10 seconds of play time, your job is to make sure that as many people remember your site as possible, and that you gain the ability to reconnect with them later, and to engage with them.

During TC50 2008, Robert Scoble blogged about Demo and TC50 websites as “sucking big time”. Knowing that he was an important influencer, we put his face on the front page of Akoha, with a speech bubble saying “If your name is Robert Scoble, click here!”, which brought the user to a small blog post explaining where we were going with our site design.

It worked. Scoble reciprocated in kind:

Being memorable is everything at an event like TechCrunch 50.  Find ways to be remembered by those who visit you.

Gathering Data

Since you’ve just launched, now is the time to start figuring out how unsolicited users react to your site. You’ve got a great lab for a few short, sweet days. Try a couple of site designs, and see which one works best.  This is where simple abtests becomes useful. Make sure you target people specifically. And turn on all the monitoring you can (without making the site slower.) It’s an excellent time to make baselines.

Google Analytics Is Not Enough

It really isn’t. For one thing, it doesn’t show you results fast enough in a flash crowd situation. Consider a more real-time analytics tool like Getclicky or Woopra, so you can see traffic as it happens — not the day after. Then find someone to watch it. While you’re on stage, or schmoozing investors, have someone back at headquarters (or whatever you call your mom’s basement) looking at the traffic.

But even if you have up-to-the-minute analytics, it’s not enough. It’s one thing to know what your users did, but chances are that you don’t even have goals set up in Google Analytics, and even if you do, you’re only measuring if people reached the goals, not why.

Measure Peoples’ Behaviors

Take a look at your site right now.  In two years, it will look completely different, because you’ll learn. What if you could speed up that clock? You can, if you measure what people do on your site and iterate rapidly.

A Web Interation Analytics tool measures where people click (or don’t), helping you figure out where your design flaws are without even asking people.  Go install CrazyEgg or ClickTale right away.  You can’t afford to miss out on this data. If you’re really serious, put Coradiant or Tealeaf in front of your site.

Ask People What They Think

People are weird. They’ll do all kinds of things you don’t expect. It’s one thing to watch what they do — but often, the real key to unlocking your business potential is to know why they did it. The simple way to find this out is to ask them. Do yourself a favor and set this up now.  Install 4Q, Kampyle, GetSatisfaction or UserVoice right away on your site.  Have the guts to ask what people like and don’t like about your site so you can fix it faster than your competitors.

And don’t just rely on tools. Be on Twitter, and be reachable. When that girl/guy in your basement isn’t watching analytics, have him or her respond to people online. When Alistair & I present at conferences, we take turns teaching and reading Twitter — fielding questions from the audience, responding to folks who couldn’t be there, and seeing if the folks in the back of the room can hear us. Do the same thing for your site.

Make Sure Your Site Can Weather The Storm

If your site is slow, people will leave.

This isn’t just a platitude: Google and Microsoft have both released empirical evidence that latency correlates with departure, and reduces conversion rates. Worse, people who had a slow experience will use the site less even when it gets faster again.

Bounce rate (the number of people that see one page and leave) may be a function of site latency. So test it! Call Keynote, Gomez, AlertSite or WebMetrics and start testing your site’s speed and availability every 5-15 minutes. Still haven’t got funding? Consider AlertFox or pingdom.  Are you at TC50, trying to load your site and it’s not working?  Not sure if it’s your computer, TC50′s Internet or your site?  Use AOL Page Test to test it remotely and find out why it’s sluggish.

Don’t forget to test the key pages and functions you want people to use, such as enrollment.

Have a Simple Call To Action

You’ll have a lot of folks on your site. Make it painfully obvious what you want them to do, whether that’s signing up, giving you permission to contact them later, trying it out, or telling friends. Don’t give them several choices — give them one. If 95% of people are just there to find out what you do, tell them. You can show them press releases or a list of your board members later. And consider tightening up your site copy, too. Less words is good.  Still not sure?  Talk to Josh Porter.  He can help.

Listen and Learn

Ultimately, you want to hear what people are saying about you elsewhere — not just on your site — and respond to the criticism and compliments.  Call ScoutLabs, TechRigy or Radian6 right away.  In most cases, all you need is a credit card and off you go.  Set up searches for keywords that matter and find conversations that concern you.  Twitter search is your friend. Google Alerts are too. There’s no better way to make a first impression than by actually being there to make a first impression.

Running The Booth?

Read this excellent post by Jason Calacanis on the subject.  It will help you avoid disillusionment when you come back from the launch party.

Learn From The Jedi Masters

Once TechCrunch50 is over, you’ll have very little time to rest on your laurels.  Chances are that your startup will die in the next year.

(Don’t die!  We don’t want you to die!)

Learn with the masters instead.  Follow Eric Ries, Dave McClure and Avinash Kaushik religiously.

Come And See Us In November (Or Call Us Before)

It’s awesome that you’ve come this far. Hopefully something in this list will help you learn from the storm, and turn the bump into a ramp. Whatever the case, once the dust settles you’ll have a lot of data to dig through. We’d love to help.

If you’re a finalist at TechCrunch50, we’re impressed that you’ve read all the way up to here.  Thanks!  We know you have so many things to do this weekend, and we appreciate that you’ve given us a few minutes of your time.

We feel for startups, we love ‘em!  So we’d like to make you an offer.  Once the next couple of weeks are done, give us a call.  We’ll spend an hour with you on the phone or online, gratis, and see if we can help you sort out your data, no strings attached.  You can reach Alistair and myself on Twitter, or simply email me directly – sean at httpd dot org.

We’ll also be doing an eight hour bootcamp called “Communilytics” at Web2Expo in New York in November on the subject of community metrics; if you’re coming, let us know.

Good luck out there, and knock ‘em dead!

- Alistair & Sean

ps; For those of you curious to know what sort of traffic the TechCrunch 08 attendees received, you’ll find the result of our findings below.

The Bump By The Numbers

When analyzing the TC50 08 finalist sites, the numbers from compete.com can be misleading.  For most sites, the traffic volume is relatively low, and therefore more prone to inaccuracies due to small sample size.  Consider that a disclaimer.  Averages were calculated between the months of October 08 and August 09 in order to avoid skewing the data as a result of the TechCrunch Bump.

It would be useless to lump all the sites in one bucket.  Grouping transaction and collaboration sites side by side is like admiring the similarities of hippopotamuses and cream puffs.

To get better insight, we must segment into the four types of sites found on the Internet.

Executive Summary

• Media sites showed relatively large traffic volumes, with unique visitor count often going up and to the right.
• Collaboration sites showed consistent traffic patterns from the month of October to august, but had no month to month average growth.
• The TechCrunch bump for transaction sites represented a small portion (8%) of the total unique visitor count they would receive for the next 11 months.
• The TechCrunch bump for SaaS portal on average represented roughly the total amount of traffic they would receive for the next 11 months.

Media Sites

These sites offer content that attracts and retains an audience. They make money from that content through sponsorship, advertising, or affiliate referrals. Search engines, AdWords-backed sites, newspapers, and well-known bloggers are media properties.

Between the months of OCT 08 (one month after TechCrunch50 ’08) and August 09 (one month before TechCrunch50 ’09), media site finalists:

• received, on average, 1,340,000 unique visitors (860,000 unique visitors with outliers removed (MIN and MAX)).
• showed peaks with a 41% increase in unique visitors, a positive influx of 148,000 unique visitors for that month.
• showed valleys with a %29 decrease in unique visitors, a loss of 171,700 unique visitors for that month. (ouch!)
• with the highest unique visitor count (MAX) showed an average monthly growth of 21.33% from 84,600 unique visitors totaling 3,530,000 unique visitors.

• with the lowest unique visitor count (MIN) showed an average monthly growth of 2.57% from 16,800 unique visitors totaling 99,600 unique visitors.

On average, the TechCrunch bump brought 57,900 unique visitors (55,300 unique visitors with outliers removed) for the month of September 08 and represented 8.55% of the total traffic they would receive in the following 11 months (8.17% with outliers removed).

Transaction Sites

A site that wants visitors to complete a transaction—normally a purchase—is transactional. There’s an “ideal path” through the site that its designers intended, resulting in a goal or outcome of some kind. The goal isn’t always a purchase; it can also be enrollment (signing up for email) or lead generation (asking salespeople to contact them), and that goal can be achieved either online or off.

Between the months of OCT 08 (one month after TechCrunch50 ’08) and August 09 (one month before TechCrunch50 ’09), transaction site finalists:

• received, on average, 272,000 unique visitors (124,000 unique visitors with outliers removed (MIN and MAX)).
• showed peaks with a 700% increase in unique visitors, a positive influx of 65,700 unique visitors for that month.
• showed valleys with a %49 decrease in unique visitors, a loss of 23,700 unique visitors for that month. (ouch!)
• with the highest unique visitor count (MAX) showed an average monthly growth of 15.49% from 113,800 unique visitors totaling 2,916,000 unique visitors.

• with the lowest unique visitor count (MIN) showed 0 growth and traffic.  (oof).

On average, the TechCrunch bump brought 35,300 unique visitors (26,600 unique visitors with outliers removed) for the month of September 08 and represented 8.83% of the total traffic they would receive in the following 11 months (6.64% with outliers removed).

Collaboration Sites

On these sites, visitors generate the content themselves. Wikis, news aggregators, user groups, classified ad listings, and other web properties in which the value of the site is largely derived from things created by others are all collaborative.

Between the months of OCT 08 (one month after TechCrunch50 ’08) and August 09 (one month before TechCrunch50 ’09), collaboration site finalists:

• received, on average, 138,000 unique visitors (86,400 unique visitors with outliers removed (MIN and MAX)).
• showed peaks with a 316% increase in unique visitors, a positive influx of 29,500 unique visitors for that month.
• showed valleys with a %63 decrease in unique visitors, a loss of 24,600 unique visitors for that month. (ouch!)
• with the highest unique visitor count (MAX) showed an average monthly growth of -1.02% from 108,000 unique visitors totaling 997,000 unique visitors.

• with the lowest unique visitor count (MIN) showed an average monthly growth of -50.70% from 1,400 unique visitors totaling 5,000 unique visitors.

On average, the TechCrunch bump brought 37,600 unique visitors (26,500 unique visitors with outliers removed) for the month of September 08 and represented 47.23% of the total traffic they would receive in the following 11 months (29.58% with outliers removed).

Software-as-a-Service Sites

These sites are hosted versions of software someone might buy. SaaS subscribers expect reliability and may pay a monthly per-seat fee for employees to use the service. Revenues come from subscriptions, and a single subscriber may have many user accounts. On some SaaS sites, users are logged in for hours every day.

Between the months of OCT 08 (one month after TechCrunch50 ’08) and August 09 (one month before TechCrunch50 ’09), software-as-a-service site finalists:

• received, on average, 62,000 unique visitors (47,000 unique visitors with outliers removed (MIN and MAX)).
• showed peaks with a 337% increase in unique visitors, a positive influx of 18,200 unique visitors for that month.
• showed valleys with a %65 decrease in unique visitors, a loss of 15,400 unique visitors for that month. (ouch!)
• with the highest unique visitor count (MAX) showed an average monthly growth of 1.52% from 26,000 unique visitors totaling 250,000 unique visitors.

• with the lowest unique visitor count (MIN) showed an average monthly growth of 64.46% from 1,400 unique visitors totaling 5,500 unique visitors.

On average, the TechCrunch bump brought 23,800 unique visitors (21,800 unique visitors with outliers removed) for the month of September 08 and represented 100.23% (!!!) of the total traffic they would receive in the following 11 months (91.81% with outliers removed).

Notes:

Big thanks to wa7iut for helping me with the stats!

Alfabetic.net was not included because it was impossible to determine what kind of site alphabetic.net is (site is down, nothing on Google cache or archive.com and the ustream demo doesn’t make it clear).

We used a combination of fotopedia.com and fotonauts.com data due to a site rebrand.

footnote.com was not included in the analysis.  By the time that they were presenting at TC50, they had 1.3million unique visitors a month, with 1.4 million unique visitors in December 2007.  Statistically (and from a TC50 perspective), footnote.com was too much of an anomaly.  We do, however, think that their site is totally cool.

We’re doing a presentation that’s excerpted from the book at DemoCamp Guelph tonight. Should be an interesting conversation; we have an “exercise” planned. Sean can’t be here (he was at Podcamp and has to get real work done after a weekend of editing the 400+ figures in the text!) but will be joining on Twitter. If you have photos from the event, or questions for Sean, we’ll be using the #CWM hashtag (for Complete Web Monitoring, the title of the book.)

One of the projects we’ve been working on is trying to create a single, comprehensive overview of the Complete Web Monitoring process. Here’s where we’re at (and an early glimpse at a poster we’re working on.)

First of all, a complete monitoring strategy includes the many questions a web analyst needs to answer:

• Web analytics (“what did they do?”)
• Web Interaction Analytics (“how did they do it?”)
• Voice of the Customer (“why did they do it?”)
• Both synthetic and real user performance monitoring (“could they do it?”)
• Community monitoring (“what are they saying?”, “who’s talking?”, and “where are they saying it?”

Any strategy also has to look at several different stages in monitoring:

• Arrival (“I visited the site”)
• Usage (“I played with it”)
• Engagement (“I’m a part of it”)
• Revenue (“I paid for it”)
• Referrals (“I spread the word”)

If these look somewhat like Dave McClure’s Pirate Metrics, it’s because he’s awesome and we borrow heavily from his thinking on startup metrics. Anyway, this PDF is a work in progress of trying to align the big questions analysts need to answer with the various stages of visitor engagement. Once we sex it up a bit, we’ll make some posters.

I’ll put the DemoCamp slides up here shortly.