Those of us who are building web apps and services inevitably ask ourselves “if I build in the cloud, which one performs the best?”.  In this scenario, the word “best” is a misnomer – what we really mean is “what cloud can perform better given the particular needs of my app?”.  Well, Alistair set out to find those answers.

Along with Webmetrics, he published research that quantifies web application performance on Amazon, Google, Salesforce, Rackspace and Terremark.  Instead of using a “one size fits all” methodology, he compared service response, network performance, CPU, and internal I/O for each cloud provider.

Here’s a quick summary of the results; you can download the full study — complete with detailed conclusions, test methodology, and even agent code –for free from Webmetrics.

No study like this has been done before – and I hope that many people will take the code, improve it, and run tests of their own.  Oh, and it’s all free.  Here’s the original post on the topic.

Enjoy :)

No related posts.

The funny thing about monitoring is that there’s a ton of data to collect, but few people know where to start. A few months ago, Steve Souders, one of the co-chairs of Velocity, asked us if we’d teach a workshop to try and fix this.

It’s a reworking of many of the things we cover in the book, plus an attempt to explain the math and reporting in an accessible way. Here’s the slide deck.

Related posts:

1. [Synthetic and Real User Monitoring] Knowing When Things Go Wrong
2. Slides from performance and KPI webinar

Alistair and I can’t wait for our Applied Communilytics session tomorrow at Web 2.0 Expo in San Francisco.  We have an all-star lineup speaking with us during the day.

We’ll hear from 5 guests in “blitz-sessions”; 15 minutes interviews.

• John Lovett, principal at Web Analytics Demystified will speak to us about the report he just released with the Altimeter group entitled “Social Marketing Analytics”
• Chris Slowe, lead developer at Reddit will speak to us about tracking and growing a community
• Kevin Weil from the analytics group at Twitter will speak to us about Twitter and the future of social measurement
• Hiten Shah, CEO of KISSmetrics will speak to us about social media metrics and optimization
• Dave McClure of Founders Fund will talk about AARRR, and separate fantasy from reality in social media measurement

We’ll also have a panel with some brilliant folks that are working to support social media analytics practitioners.  We’ll talk about the present and future of social media, concentrating on “quick wins” for analytics geeks.  The panelists include:

• Ryan Kuder, VP of Marketing at Biz360
• Erin Hunter, EVP at comScore
• Eric Feinberg, Industry Director at ForeSee Results
• Ryan Holmes, CEO at HootSuite
• Matt Langie, Sr Dir of Product Marketing at Omniture

The schedule for our day is as follows:

You still have time to register! Here’s a 25% off discount code if you’d like to attend: websf10ac25

We hope to see you there!

No related posts.

I’m in a few cities in the US presenting an overview of Complete Web Monitoring for my old alma mater, Coradiant and their partner, Dynatrace. It’s a 45-minute recap of the concept of holistic monitoring, looking at the four kinds of website and linking together disparate sets of data with a particular focus on performance monitoring. Slides here:

If you’re in one of the cities, sign up and stop by. Next stop: Interop, then Web2Expo San Francisco for the Communilytics workshop!

Related posts:

1. Applied Communilytics – 10 Speakers & Fresh Material

Disclaimer: I’m one of the nominees for the WAA board of directors. Shouldn’t really matter, but consider yourself warned.

This year, 21 nominees are running as nominees for five board of director positions – 1 in Europe and 4 in North America.

I’m running for three major reasons:

• Social Media Measurement: Let’s make sense out of them.  I co-wrote a book on the subject in 2009 with Alistair Croll. I’ve talked to hundreds (thousands?) of practitioners / enthusiasts about the subject. The WAA needs to pursue it’s standardization, advocacy and education in the field.  Yep. I can help that.
• Lean Startup Analytics: It’s time we give more love to the massive startup ecosystem. Startups have completely different needs than companies in the enterprise space. By appealing to them, we can grow our membership base considerably and help educate those that will turn into the mammoth companies of tomorrow.  Think of it as “educating our young”.
• Non-Profit Analytics: Much of the efforts put into strengthening the startup base can equally help non-profits.  By passing along educational material targeted to non-profits, we can empowering them with stronger abilities to  understand where their efforts are successful, and where they’re not.

During the day time, I, eat, poop and sleep data convergence.  We need to be more inclusive of non-traditional streams of measurement. This will ultimately help us grow our member base, and make the organization more relevant in today’s “big data” environment.  This is the central thesis in Complete Web Monitoring.

There are 12 people on the board of directors and the WAA has over 1500 members (updated number from 5000, thanks Eric Peterson).  We have the resources, collective intelligence and drive to make strides in each of these areas.  We simply need a bit of support, encouragement, and direction to make these a reality.  Aaaaand, that’s why I came to the party :).

But frankly, as far as the vote is concerned, I’m confused. It isn’t how I imagined things being. I never imagined that there would be as many existing board members running as there are positions.  The people I’m “running against” aren’t people I want to run against.  I’m supposed to be running against Jim Sterne, the founder of the WAA?

A big part of me wants to give Alex, Dennis, Jim, June and Vicky a second BoD term. Jim puts it so eloquently:

“Aside from being a major cheerleader for the cause, I bring the organizational memory which will help the next Board and our new Executive Director understand why certain decisions were made in the past.”

Another part of me wants to encourage voters to pass the torch on to new candidates armed with fresh ideas and new perspectives.

• John Lovett, analyst extraordinaire, brings an understanding and tenure in the web analytics industry that few nominees can rival.  I encourage the WAA to take the opportunity to work with him closely over the next two years.
• Brendan Hart brings executive metrics knowledge from National Geographic that can greatly benefit our organization from a practitioner point-of-view.
• Eric Feinberg never ceases to amaze me with his candor, knowledge and ability in the field.  He is a great person to talk and work with and would be a big asset to the WAA.
• Steve Jackson, who wrote the book Cult of Analytics, and pushes hard for awareness of the WAA outside of North America.
• And . . . well . . . I’m running and all. :)

I’m not sure which way to go. No matter what, I feel like the Web Analytics Association will gain some great directors, and lose some great candidates.

I’ll probably end up voting for a mix of organizational memory and fresh meat. Either way, the decision will be very hard.

For all of you planning to vote (and even those that aren’t!), I’d love hear your thoughts – either here or on your own blog (if you blog about it, let me know – I’ll add all WAA related posts to the end of this one). By encouraging discussion, I hope that it will help us all come to a conclusion that’s best for the association.

See you on the ballot.

PS: You can see a nominees in action on Twitter by following the Twitter list or finding their individual accounts here.

Here’s a list of blog posts related to the Web Analytics Association Board of Directors vote:

• Vote in the Web Analytics Association Election! by Larry Freed on FREEDyourMind
• WAA Elections: Accountability, Inclusion, and Value by Eric T Peterson on Web Analytics Demystified
• Vote Lovett for the WAA Board of Directors by John Lovett on Web Analytics Demystified
• Big Election Day in Web Analytics Citizenship by Jacques Warren on WAO Marketing

Related posts:

1. Web Analytics Association Board of Director Nominee Twitter List
2. Beth Kanter and Non-Profit Analytics

If you haven’t heard yet, the Web Analytics Association is an organization that’s promoting web analytics.  It does so by passing on information to the public, by providing access to best practices, and ultimately connecting individuals, vendors, practitioners and consultants in the field.  This year, the WAA received nominations for their board of directors.  It is accepting 5 positions – 4 based in North America and 1 based in Europe to compliment their existing board.

I’ve put together a Twitter list of all nominees running for the 5 Web Analytics Association board of director positions.  To follow the nominees, simply click on this link, and hit the “Follow this list” button.  Even better, simply add this list if you use Seesmic Desktop, TweetDeck, CoTweet, HootSuite or any other Twitter application.

If you’re interested in following the nominees individually, I’ve included their Twitter accounts below.  Enjoy!

Related posts:

1. Thoughts on the 2010 Web Analytics Association Board of Directors Vote
2. Beth Kanter and Non-Profit Analytics

There’s a known weakness in browsers which we wrote about in the book. Every time we talked with someone about it, they’d ask us why we didn’t start a company that took advantage of the loophole, and the answer was, well, it’s creepy. The loophole basically lets you see where else your visitors have been on the Internet. Well, it’s now out in the open, in two forms: Beencounter, and Haveyourfriendsbeenthere.

To be perfectly clear, the site won’t show you everything your visitors surf–just whether or not they’ve been to a set of sites you define. Here’s how it works:

1. You decide what sites you’d like to find out about
2. You embed these sites in a hidden portion of the page
3. When a visitor loads the page, the sites that are visited are marked a:visited in the page’s CSS
4. The Javascript in the page can then grab this property of each link and send it back to you

Knowing where a visitor has been can be used for all kinds of things. For one thing, using just a few sites you can guess the visitor’s gender with a good degree of confidence–resulting in more targeted advertising. This isn’t a new idea (it’s been discussed in terms of browser history before). You might also offer a discount to visitors who’ve already checked out your competition.

Haveyourfriendsbeenthere takes advantage of the obfuscation from a short URL to hide what it is, meaning many people will click on it inadvertently. There’s no easy way to fix this without breaking a lot of the history functions that we use when browsing (one user on Reddit pointed out that this flaw has been around since 2002 and there are sites that show your surfing history already).

We figured it was worth talking about it more openly since these two services are likely to make it a pretty mainstream practice, particularly among sites that benefit from demographic targeting.

BTW, clearing your browser history or surfing in anonymous mode will hide your behavior from such tools.

No related posts.

Beth Kanter’s 53rd birthday is today.  Of the many reasons why we’re big fans of Beth, she’s pioneering the concepts of web-based analytics for the non-profit / charity sector.  If you haven’t seen her blog before and you want to deep dive in her thoughts on metrics, start here.

To celebrate her 53rd birthday, she’s using social media to incite change in the world by sending 53 Cambodian children to school.  Here’s the full description of her birthday wish.  Have a few bucks laying around?  Help her out here.  It’ll help kids go to school in Cambodia!  How cool is that :).

It gets better.  She’ll write about the lessons she learned during the campaign (just like we did for the Beers for Canada), and share insights on the metrics and measurement tactics she used to determine what worked and what didn’t.

As always, Beth continues to be a rock solid voice in the world of non-profits & web.  She’s on our A-list.

Happy birthday Beth!

No related posts.

Recently, I was in Israel for a cloud computing conference and some meetings with local VCs. The folks at Gemini, a VC firm, organized an evening with their portfolio CEOs to discuss lean analytics for startups. I concluded the presentation with a list of metrics that a web-based startup should track. I guess they were the right questions; at the end of the evening, Guy Horowitz, my host for the event, said,

“I feel bad for the CEOs of my portfolio companies that aren’t here. Their next board meeting will be miserable.”

Not measuring the right things can be fatal. And VCs are in the business of separating the soon-to-be-dead from the fledgling successes. There’s nothing quite as good at doing this as the cold, hard light of analytics. So here’s the list, with a slide deck and some examples.

Message reach

For many startups, the web is their main marketing medium. Because everything online can be tracked, it’s the perfect platform for accountability. All too often, however, we look at metrics like traffic to the site and number of followers. These don’t matter. What matters is your ability to get a message out to people effectively. If you have a cheap way of reaching many people, that’s a valuable asset.

• Your viral coefficient is perhaps the best example of this. Simply put, it’s a measure of how a message will propagate. CEOs and VCs need to know, on average, for each message they hear, how many people does someone tell?

• One measure of this on open social networks is amplification. How well are messages being amplified? This varies by social network: On Twitter, it’s the number of Retweets; on Reddit, the number of upvotes; on Facebook, the number of people who become fans.

Infrastructure health

Many companies overlook infrastructure health, despite considerable recent evidence that page load times strongly influence conversion rates. For an investor, unhealthy infrastructure is often a sign that the startup lacks operational discipline and can point to deeper problems — hacks, kludges, and the kind of duct-tape improvisation that’s fine for the prototype but won’t do in production.

• Performability, which is a combination of performance (latency) and availability (uptime) is usually enough here. But what you measure matters a lot. You need to track the health of key steps in the business process, such as the enrollment loop or the invite process. If you’re a Software-as-a-Service company, you need this data to resolve disputes with subscribers and prove you’d met any Service Level Agreements you’ve commited to.

Market sentiment

You want to know what your market thinks of you. There are many ways to grab this information, but they boil down into two major categories.

• Voice of the Customer surveys ask visitors what they think when they’re on your site.  You can get it from companies like iPerceptions, Kampyle and Foresee. This is crucial in the early stages of a startup, when you’re trying to decide what business you’re in; it’s also vital when evaluating new features, and can act as a safety net when something breaks because a frustrated visitor has somewhere to vent.
• Community sentiment is a combination of search engines and natural language processing. Several startups such as Scoutlabs try to understand what’s being said, where, and how people feel about a particular topic. It’s a hard task to do well because computers don’t understand sarcasm, but knowing when people are talking trash about your brand online is essential.

User engagement

How engaged are your visitors? Ask yourself: How many accounts have you signed up for, then abandoned? If you were to search your inbox, you’d probably find dozens of password confirmations you’ve long since forgotten about. If you’re running a startup, then the chances are good that you’ll be forgotten too. This is an incredibly painful thing to look at. But if it doesn’t make you want to kill yourself, it’ll make you stronger. You can try out new strategies to engage your community and motivate visitors to return, and see if they’ve worked.

• The best way to show this is to measure the time since the last visit by individual user, on a histogram. Here’s an example.

We like this approach because you can see, at a glance, whether people are becoming more or less engaged, as well as how big your “dead pool” of visitors is. We usually map the two on separate axes because your “dead pool” is probably very big — bigger than you expect. Sorry about that.

Lean analytics

Too often, the web data shared at board meetings doesn’t match the business plan. In the business plan of record, you have some assumptions — conversion rates, content a visitor will share, friends they’ll invite. You need to identify these goals, then report them over and over again, with trends.

• Core goals. What are the 3 or 4 assumptions on which the business hinges? If you can’t explain these — and if you don’t lie awake thinking about them every night — find another person to lead the company. This should be the most important thing in your business. Always know what core tasks  you’re hoping your visitors will accomplish, and why they correlate to business growth. If you want to change one of them, make sure your investors all agree to the change, because you’re effectively changing the business model in which they invested.
• Extended funnel abandonment. Each of these goals has a conversion funnel of some sort related to it. Often, the start of that funnel is off on a social network, where someone hears about your site or product. During the achievement of a goal, a visitor probably has to do things that aren’t on your site — opening an email to confirm an address, for example, or authorizing the site to link to Facebook. Each of these steps towards the goal must be tracked. Check out the Product Planner website for some great examples of these steps, along with metrics to watch for each one.
• Movement towards or away from business goals. At every board meeting, you should know what changes to the product moved you towards or away from a goal. If your goal is to minimize calls to tech support, then what changes helped with that and what didn’t? If the startup can’t answer this question, it’s because it’s not experimenting to see which things affect business goals, and that’s a very bad thing.
• Changes to the funnel. Seeing the funnel is useful; seeing it compared to the past period is essential. If you have an extended funnel, and you can see what changed (“email confirmation bounce rates climbed” or “fewer people clicked on the bit.ly link in Twitter”) then you know where to focus your efforts in the coming weeks.

Business sustainability

Once, companies had to decide whether or not to shut down because they were hemorrhaging money. Today, many web businesses can “hibernate”, spending only a few hundred dollars a month on cloud-based hosting while they wait and try to grow organically. That means there are three trajectories: growth, failure, or sustainable burn. Some key metrics will tell you where you stand.

• Cost per engaged visitor is simply the operating costs of the site divided by the number of visitors. It’s how much it costs to provide service to someone. This is a great metric because you can track it month over month, and understand the impact of hiring or capital expenditure on the site. It had also better be smaller than the revenues and conversion rates you’re seeing, or you’re doomed.
• Peak to average ratio. All sites experience traffic spikes. How much busier is the site at busy times — weekends, seasonal holidays, news events, and so on? What premium do you pay to be able to handle those peaks? And can you decommission well? Lots of people worry about scaling up; fewer remember that elasticity is the ability to shed infrastructure quickly and keep things efficient.
• Minimum sustainable burn. If you had to hibernate, what would the steady-state cost of keeping the site running be? This includes salaries, hosting costs, essential services, and so on. Think of it as a disaster plan. If you’re not seeing adoption, and you’re running out of runway, this is the most important number in the company.

It was a great discussion with Gemini’s CEOs, and a good opportunity to discuss what things other companies are tracking. The deck from the Gemini event is available on Slideshare and embedded below.

If you’re a VC or a group of entrepreneurs who’d like to know more about lean startup analytics, bug Sean or myself on Twitter. We’d love to chat.

No related posts.

Alex Bowyer (@alexbfree) is a research analyst at Bitcurrent in Montreal, where he blogs about emerging technologies and their social impacts, and co-organizes events such as Bitnorth and Enterprise Cloud Summit. He is passionate about using computers to solve human problems in new ways, and all the things that encompasses – user-centric design, productivity, human-computer interfaces and exploring social trends. Before that he worked at IBM UK, specializing in Voice systems, Java and information management.

In this post, Alex shows us that sometimes, attracting pageviews isn’t rocket science, which you might be forgiven for thinking if you follow this blog regularly:

Unlike my colleagues Alistair and Sean, I’m no analytics expert. But like all bloggers and social media enthusiasts I have an interest in sharing ideas about technology and society, and getting those ideas out to as many  people as possible.

I’ve been exploring ways to get more traffic to my personal blog, and yesterday stumbled upon something quite remarkable. In one day I was able to achieve 2,579 new pageviews, a 1,176% increase in traffic. And all it took was about 30 minutes of effort.

I’ve been using StumbleUpon’s su.pr as my preferred link shortener for some time, because as well as shortening the URL it allows me to contribute those links to my existing StumbleUpon social bookmark feed while also generating some useful stats. I noticed recently that it makes it very easy for you to “thumb up” and review your shortened links, adding them into the StumbleUpon ecosystem:

[For those who are unfamiliar, StumbleUpon is a social bookmarking service that differs from Digg, Reddit or Delicious in that allows you to click a toolbar button and "Stumble" to randomly selected pages, bookmarked by others, from across all the topics you specify an interest in.]

I also noticed that those links I had submitted to StumbleUpon had gained significantly higher amounts of traffic than those that hadn’t. So as an experiment, I went through every single blog post on my blog and “thumbed-up” the site, assigned it to some appropriate tags and stored it in the best fitting topic. (This last step is vital to maximize the chance of interested eyes landing on your page.)

I waited a day, and I think the results speak for themselves:

What was even more satisfying was to see that these were not just new visitors, but they engaged. Most of the new visitors went on to read other content on my blog, as you can see from the reduced bounce rate,

as well as 1,084 new goal conversions (albeit fairly simple goals such as the amount of time spent on the site):

I think that the reason this experiment was so successful is that people clicking that Stumble button are actively seeking “something interesting” – and are very open to new content and ideas, more so than the average web user. This coupled with the fact that StumbleUpon only directs people to pages for topics they have said they are interested in, results in not just more eyeballs, but the right sort of eyeballs.

I would be fascinated to see this approach used on a site with more measurable goals such as product sign-up or purchases*.

* Disclaimer: Stumblers can rate you “thumbs down” as well as up. Your site reputation may be at risk if you don’t keep providing value to those who rely on it.

Related posts:

1. Guest Post: How much is enough when it comes to Voice of Customer?