Recently, I was in Israel for a cloud computing conference and some meetings with local VCs. The folks at Gemini, a VC firm, organized an evening with their portfolio CEOs to discuss lean analytics for startups. I concluded the presentation with a list of metrics that a web-based startup should track. I guess they were the right questions; at the end of the evening, Guy Horowitz, my host for the event, said,
“I feel bad for the CEOs of my portfolio companies that aren’t here. Their next board meeting will be miserable.”
Not measuring the right things can be fatal. And VCs are in the business of separating the soon-to-be-dead from the fledgling successes. There’s nothing quite as good at doing this as the cold, hard light of analytics. So here’s the list, with a slide deck and some examples.
Message reach
For many startups, the web is their main marketing medium. Because everything online can be tracked, it’s the perfect platform for accountability. All too often, however, we look at metrics like traffic to the site and number of followers. These don’t matter. What matters is your ability to get a message out to people effectively. If you have a cheap way of reaching many people, that’s a valuable asset.
- Your viral coefficient is perhaps the best example of this. Simply put, it’s a measure of how a message will propagate. CEOs and VCs need to know, on average, for each message they hear, how many people does someone tell?
- One measure of this on open social networks is amplification. How well are messages being amplified? This varies by social network: On Twitter, it’s the number of Retweets; on Reddit, the number of upvotes; on Facebook, the number of people who become fans.
Infrastructure health
Many companies overlook infrastructure health, despite considerable recent evidence that page load times strongly influence conversion rates. For an investor, unhealthy infrastructure is often a sign that the startup lacks operational discipline and can point to deeper problems — hacks, kludges, and the kind of duct-tape improvisation that’s fine for the prototype but won’t do in production.
- Performability, which is a combination of performance (latency) and availability (uptime) is usually enough here. But what you measure matters a lot. You need to track the health of key steps in the business process, such as the enrollment loop or the invite process. If you’re a Software-as-a-Service company, you need this data to resolve disputes with subscribers and prove you’d met any Service Level Agreements you’ve commited to.
Market sentiment
You want to know what your market thinks of you. There are many ways to grab this information, but they boil down into two major categories.
- Voice of the Customer surveys ask visitors what they think when they’re on your site. You can get it from companies like iPerceptions, Kampyle and Foresee. This is crucial in the early stages of a startup, when you’re trying to decide what business you’re in; it’s also vital when evaluating new features, and can act as a safety net when something breaks because a frustrated visitor has somewhere to vent.
- Community sentiment is a combination of search engines and natural language processing. Several startups such as Scoutlabs try to understand what’s being said, where, and how people feel about a particular topic. It’s a hard task to do well because computers don’t understand sarcasm, but knowing when people are talking trash about your brand online is essential.
User engagement
How engaged are your visitors? Ask yourself: How many accounts have you signed up for, then abandoned? If you were to search your inbox, you’d probably find dozens of password confirmations you’ve long since forgotten about. If you’re running a startup, then the chances are good that you’ll be forgotten too. This is an incredibly painful thing to look at. But if it doesn’t make you want to kill yourself, it’ll make you stronger. You can try out new strategies to engage your community and motivate visitors to return, and see if they’ve worked.
- The best way to show this is to measure the time since the last visit by individual user, on a histogram. Here’s an example.
We like this approach because you can see, at a glance, whether people are becoming more or less engaged, as well as how big your “dead pool” of visitors is. We usually map the two on separate axes because your “dead pool” is probably very big — bigger than you expect. Sorry about that.
Lean analytics
Too often, the web data shared at board meetings doesn’t match the business plan. In the business plan of record, you have some assumptions — conversion rates, content a visitor will share, friends they’ll invite. You need to identify these goals, then report them over and over again, with trends.
- Core goals. What are the 3 or 4 assumptions on which the business hinges? If you can’t explain these — and if you don’t lie awake thinking about them every night — find another person to lead the company. This should be the most important thing in your business. Always know what core tasks you’re hoping your visitors will accomplish, and why they correlate to business growth. If you want to change one of them, make sure your investors all agree to the change, because you’re effectively changing the business model in which they invested.
- Extended funnel abandonment. Each of these goals has a conversion funnel of some sort related to it. Often, the start of that funnel is off on a social network, where someone hears about your site or product. During the achievement of a goal, a visitor probably has to do things that aren’t on your site — opening an email to confirm an address, for example, or authorizing the site to link to Facebook. Each of these steps towards the goal must be tracked. Check out the Product Planner website for some great examples of these steps, along with metrics to watch for each one.
- Movement towards or away from business goals. At every board meeting, you should know what changes to the product moved you towards or away from a goal. If your goal is to minimize calls to tech support, then what changes helped with that and what didn’t? If the startup can’t answer this question, it’s because it’s not experimenting to see which things affect business goals, and that’s a very bad thing.
- Changes to the funnel. Seeing the funnel is useful; seeing it compared to the past period is essential. If you have an extended funnel, and you can see what changed (“email confirmation bounce rates climbed” or “fewer people clicked on the bit.ly link in Twitter”) then you know where to focus your efforts in the coming weeks.
Business sustainability
Once, companies had to decide whether or not to shut down because they were hemorrhaging money. Today, many web businesses can “hibernate”, spending only a few hundred dollars a month on cloud-based hosting while they wait and try to grow organically. That means there are three trajectories: growth, failure, or sustainable burn. Some key metrics will tell you where you stand.
- Cost per engaged visitor is simply the operating costs of the site divided by the number of visitors. It’s how much it costs to provide service to someone. This is a great metric because you can track it month over month, and understand the impact of hiring or capital expenditure on the site. It had also better be smaller than the revenues and conversion rates you’re seeing, or you’re doomed.
- Peak to average ratio. All sites experience traffic spikes. How much busier is the site at busy times — weekends, seasonal holidays, news events, and so on? What premium do you pay to be able to handle those peaks? And can you decommission well? Lots of people worry about scaling up; fewer remember that elasticity is the ability to shed infrastructure quickly and keep things efficient.
- Minimum sustainable burn. If you had to hibernate, what would the steady-state cost of keeping the site running be? This includes salaries, hosting costs, essential services, and so on. Think of it as a disaster plan. If you’re not seeing adoption, and you’re running out of runway, this is the most important number in the company.
It was a great discussion with Gemini’s CEOs, and a good opportunity to discuss what things other companies are tracking. The deck from the Gemini event is available on Slideshare and embedded below.
If you’re a VC or a group of entrepreneurs who’d like to know more about lean startup analytics, bug Sean or myself on Twitter. We’d love to chat.
No related posts.
