Don’t Be Fooled By Last-Click Analysis Of Social Media

Forrester recently published a report entitled “The Purchase Path of Online Buyers.” Normally I’m a fan of Forrester’s reports, but this one left me scratching my head.

The report looks at transaction data from 15 clients of a marketing agency (let’s ignore that built-in bias, and convenient product placement in the recommendations, for the sake of this post) to draw conclusions about buyer behaviour including:

  • Most buyers do not arrive at a site directly — they come from search or other marketing activities (fair)
  • Last-click measurement is insufficient – it works for email and search but other tactics receive insufficient credit “as they are typically early in the research funnel and are followed by visits to search engines or email” (fair)
  • Email was effective during key promotional dates — not surprising, as retailers engage in heavy email drives during those times (think of Thanksgiving or Boxing Day)
However, one conclusion stood out to me:
“Hope for the best, but expect the worst with social.”

(Not surprising that it caught my eye, huh?)

The more I thought about, it the more I was left confused at Forrester’s characterization of the data, especially given the earlier warning about last-click measurement.

Where to begin?

No detail in the methodology

The report doesn’t actually give any detail as to the form the data that was used took. How did they track referrers, especially those two levels deep? Could they identify traffic that came from mobile social apps or from popular desktop apps such as TweetDeck, etc? From the methodology it sounds like they used the agency’s own models to estimate the data, but really we have no idea.

Zero detail on actual social media activities

At no point in the report does it say that (a) the companies were engaged in any sort of social media activities, (b) the form that those activities took (c) the scale and reach that those activities had, or (d) the quality of those activities.

How are we supposed to just agree with the conclusion that social media drove minimal sales if we don’t know what form that social media took, if any? It’s like saying “media relations drove minimal sales” without saying whether the organization actually did any media relations.

Hell, the numbers could actually be a good thing - if I wasn’t engaged in any form of social media but it still drove 2% of my sales, that might actually be a sign that I should begin to invest in it.

Social media objectives vary

In my opinion, brands make a mistake when they consider social media as purely conversion-driven tools. Social media provides numerous potential benefits besides end-of-funnel conversion, including:

  • Long-term brand-building
  • Top-of-mind awareness
  • Improved customer service and retention
  • Market intelligence and insights

Social media isn’t just bottom-of-the-funnel

Related to the last point, companies should consider the average person’s mindset when they’re using social media.

When you’re using Facebook or Twitter, for example, are you often looking for a link to take me to the best place to purchase something? Probably not. If you’re in a purchase-focused mindset, you’re more likely to be looking for reviews or recommendations from other people.

Once you receive them, maybe you’ll dig around for product information on the recommendation, then look for somewhere to buy it.

For example, today I asked Twitter whether I should buy a LiveScribe pen so I could capture my notes in Evernote:



Sure enough, I got a bunch of points of view in return:

Note: None of those responses offer a way to click through to purchase. Still, even though these results affected my likelihood to purchase, if I were to purchase the pen (which I may), a search engine would likely get the credit — even though search has done nothing to influence my actions up to this point — as I would use Google to find the company’s website.

Forrester seems to be ignoring its own warning that tactics other than email and search may be under-attributed, and passing judgement without fully considering the context.

Social media isn’t one-dimensional

This Forrester report focused on online buying behaviour, so it might seem harsh to criticize it for its single-minded focus. That’s the problem, though — when you’re dealing with a complex, multi-functional set of tools like social media, considering a single dimension in isolation from the others risks writing-off broad benefits across the organization.

This only reinforces the need for a broader focus than an eight-page report can produce – one looking at the effects of integrated, multi-disciplinary approaches to social media. Analyzing an inherently integrated, multi-dimensional set of tools in any other way leads to an incomplete picture.

(Image: talltomz)

7 comments
Allenyoung
Allenyoung

Hello, The only way to avoid criticism is…not to upset people. Bugger, that’s going to be tough. Just ask the folk over at Gap Towers. Heeding the boundaries of the consumer comfort zone pretty much kills all chance of innovation, development, edgy marketing campaigns, or even fun. I probably wouldn’t be allowed to write this drivel for starters. ........... Allen young

Aaron Bradley
Aaron Bradley

Great post.  However, I disagree when you say - in your analysis of the report - that "[l]ast-click measurement ... works for email and search."  Organic search is notorious for being underrepresented when last-click metrics are used. In fact, organic search shares a lot of the same analytical challenges as social, because it is so high up in the information discovery chain.  In the realm of ecommerce, most people start what will end up as a purchase with a web search:  between that initial search and the purchase the customer may visit any number of sites that will end up with search referrers being severed from the purchase in analytics.

Niloofar Mirani
Niloofar Mirani

 Dave, I agree with your comments wholeheartedly. Like @baznet:disqus I am surprised that these errors weren't caught by others in the social media or marketing research industries. When making conclusions, especially about hot topics like social media, it is important to be sure your research methods are accurate and can be defended.. Thank you, as always, for an insightful post!

Ruth Bazinet
Ruth Bazinet

Dave - I am really surprised that some of your observations on this report haven't been written about already. Where are we as industry that social folks weren't all over this report when it was published in March? The report opens more questions than it answers. You are dead on with your criticism, particularly on the social strategy (and said effectiveness) of the companies involved in the study.  The processes of measuring social strategy effectively is moving as fast as social itself so I have sympathy for the researchers. However, that consideration should have been incorporated into this study. Simple as that.  As silly as this may seem, I don't think all online marketers are speaking the "social" language. I hope that this study doesn't help validate that self-imposed barrier for those people.

davefleet
davefleet

I think there's a difference between avoiding criticism and providing constructive feedback. The latter is the difference between "this is crap" and "here are three ways I might have done this differently", which is an important one. Also, guessing you meant to post this comment on a different post...

davefleet
davefleet

Given that that part of the post was just summarizing the report's conclusions, if that's the case then we can add that to the list of issues with the document!

Trackbacks

  1. [...] year I was sceptical about Forrester’s 2011 report given that the data was taken from the clients of a single marketing agency – and frankly [...]