We had a good discussion about performance and its impact on KPIs like analytics and conversion with Strangeloop this week. Here are the slides, available for download or viewing, on Slideshare.

Do faster web pages mean better business? Definitely. We’ve seen hard evidence from major web operators like Shopzilla, Google, and Microsoft. But what about other websites? How big an impact does performance optimization have on the business metrics of a typical media or e-commerce site?

Here’s some concrete data on how reducing latency changes the key metrics, such as bounce rate, pages per visit, conversion rate, and shopping cart amount. It’s a pretty detailed discussion, but it if you want to understand the ROI of improving web performance on your site, dig in. If you want to read this more easily, here’s a PDF.

Enough theorizing: Page latency affects your business

Years ago, the need for web performance was anecdotal. Researcher Mihaly Csikszentmihalyi had, years before, shown that human beings are more engaged, and more likely to enter “flow states”, when response to their actions is immediate (see this article for more details on flow and web optimization.) But while it seemed like a good idea to make a page load quickly — and companies like Zona Research made headlines with their 8-second rule — we didn’t have any empirical evidence to that effect.

There’s no longer any debate. There’s reliable, reproducible evidence that web page latency is directly tied to the bottom line. At Velocity, Microsoft, Google and Shopzilla made this abundantly clear in a series of awesome presentations: detailed, controlled testing proves that slower pages hurt the bottom line. In Google’s case, adding delay reduces the average number of searches a visitor does each day even after the delay is removed.

Microsoft, using data from Bing, showed that slow pages affect other KPIs:

In other words, if your website is slow you’ll get:

• Fewer search queries per user
• Less query refinement
• Less revenue per visitor
• Fewer clicks, and lower satisfaction
• A longer time for visitors to click something
• Fewer searches per day
• Lower search engine rankings

Ouch.

There’s a great writeup of all this at Artzstudio, and we covered Marissa Mayer’s talk on Bitcurrent. My only complaint was that the sessions all left me with questions about how this applies to smaller, more targeted sites.

So what about sites we mortals run?

Generally speaking, there are a few KPIs that really matter. These measure attention (how many people find out about you), engagement (how much they interact with your site), and conversion (whether they do what you wanted, and how much you benefit from it.) Here’s a simple view of some of those KPIs. Different KPIs matter as visitors turn into customers, buyers, or enrolled users.

In this diagram, inbound traffic either consists of returning or new visitors. Some of those visitors leave (“bounce”) immediately, while others continue on to other pages, spending time on the site, creating content, and viewing ads. On an e-commerce site, a subset of those visitors convert (by doing something we want them to) and there’s a value to those conversions.

The question, then, is how does a faster page load time affect metrics such as these?

Comparing performance optimization in analytics

At Interop Las Vegas, I had dinner with the guys from Strangeloop Networks (disclaimer: I have friends who work there). They make a web acceleration appliance that speeds up page load times. It occurred to us that they were in a perfect position to tie page performance to analytical results, because their appliance could actually modify pages on their way out, on a visitor-by-visitor basis. So if they only optimized some of the visits, and marked them as such, they could later compare the business performance of optimized and unoptimized segments.

The Strangeloop guys ran sprinted with this idea, testing a couple of different types of site, including media and transactional businesses. Here’s what they found, which they’ve agreed to share with us provided we keep certain details confidential.

Total visits

The system was set up to optimize half of the visits and leave the other half untouched. But when they analyzed the results, that’s not how things looked in Google Analytics, where significantly more optimized sessions were recorded. In all, roughly 14,000 visits were segmented within the analytics package; but the number of optimized sessions that were recorded was significantly higher.

This may be due to problems with the analytics scripts running on slower connections, or it may be a sign of increased page abandonment before analytics has a chance to load. Either way, it’s worthy of more study.

Percent of visits from new visitors

Optimization seems to have an effect on the number of new visitors to the site, too, though it’s not clear why this happens.

We’d expect that since new visitors’ browsers have more to load — there’s nothing in their cache yet, and so page load times are higher — there would be fewer aborted visits for new users, which in turn would increase the relative number of new visitors to the site. But that’s not what happened.

Bounce rate and optimization

The next metric we looked at was bounce rate, that is, how many visitors left quickly from the first page they saw. Faster pages delivered a lower bounce rate than slower ones.

This seems intuitive, since when a page loads quickly you’re less likely to leave out of frustration. That extra percent of visitors who stick around turns into more opportunities to sell something or otherwise engage a visitor.

Engagement: Pages per visit and visit duration

Optimization also affects the number of pages a visitor views. When the site is slow, people read fewer pages.

It’s not clear from the pages-per-visit metric alone whether this is because people spend a finite time on a site, and simply get bored after that time, If that were the case, we’d expect a faster-loading page to result in more pages per visit simply because more pages could be loaded before the visitor got bored. A second KPI, Average time on site, clarifies this. Optimized visitors to the site spent 7 minutes more than unoptimized visitors. So it’s not so much the pages per minute of time on the site, but the actual number of minutes, that increases.

This is great for visitor engagement; if you’re running a media site, it also means a chance to deliver more impressions and make more money per visitor.

Impact of optimization on e-commerce results

But what about e-commerce and retail? Strangeloop instrumented a second site in the same way. The beauty of tagging visits up front is that the actual business outcome of that optimization can finally be quantified. In this case, optimization resulted in a 16.07% increase in conversion rates and a 5.50% increase in average order value.

We can’t share the actual conversion rate and order value amounts, but what these numbers do is allow you to actually quantify the ROI of a performance improvement investment.

Want to learn more?

There’s lots of good data behind these results, which we’ll be looking at in more detail in a Webinar on October 8 at 2PM EDT (you can sign up online at bit.ly/perfwebinar). Strangeloop will also have some data on how much performance improvement visitors experienced by then, and you can ask Hooman Beheshti (their VP of Products) and I questions about the experiment if you want to know more.

Some caveats:

I’m always wary of presenting vendor-specific data, because we try to remain impartial. Strangeloop isn’t paying me to talk about this, and I decided to cover it because it’s useful to the web monitoring community and I asked the question in the first place. I’ve reviewed the information fairly closely and have good reason to trust Strangeloop (for one thing, their VP of products, Hooman Beheshti, is a sometime contributor to Bitcurrent and an expert on web performance who reviewed Complete Web Monitoring and gave us detailed feedback.)

Instrumentation happens as follows:

• Every visitor who requests a page gets a segmentation cookie regardless of who they are. Since Strangeloop’s technology takes advantage of certain features in more modern browsers, not every visitor who is accelerated will benefit from the same performance improvements.
• The numbers reported in the analytics package (Google Analytics) are a result of the segmentation cookies seen, which is tied to how many visitors’ browsers made the analytics request.
• The reporting Javascript within the page sometimes can’t find the cookie that was set (this happens about 5% of the time). This could be a consequence of security restrictions, browser limitations, and so on. However, the “cookie not found” errors occur relatively evenly across optimized and unoptimized visitor, so they don’t distort the numbers.
• Google Analytics’ IP filtering was used to block out internal users, which might distort numbers too.