Posts Tagged ‘statistics’

Lies, Damned Lies and Mis-understood Statistics

11.09.21

Earlier this week, All Facebook featured a post on a report by DDB and OpinionWay examining the reasons that Facebook users “unlike” brand pages.

The key findings, as reported by All Facebook (the report appears to have been taken down from SlideShare, so I can’t link to it):

  • The brand was no longer of interest to me (49 percent);
  • The information available was not interesting (46 percent);
  • Information was published too often (36 percent);
  • The brand published information I did not appreciate (27 percent)
  • Information was not published often enough (14 percent).

Interesting, useful data.

This follow-up chart in the post, however, is next to useless.

Why is this chart useless?

Because the sample size is too small for this kind of segmentation.

The original data is useful because the analysis is conducted at an aggregate level, over 630 respondents. At that sample size, we’re looking at a 3.9% margin of error at a 95% confidence level. That means, while there may be some variation among the top results, they’re useful at a high level.

Dig down to a country level like the chart above, though, and things start to fall apart. With a sample size of 78, given the number of Facebook users in the United States (155,746,780 according to Facebook), the margin of error for the US numbers is over 11% at that same confidence level. It’s not just the US, either – the margin of error for the France numbers is over 8.5%. Despite this, there’s no mention of these details on the post or comments; just an assumption that the numbers are correct.

This is a great example of why I think math is a critical skill for PR professionals.

PR pros need to understand the difference between valid statistics and invalid ones, so they can take advantage of useful information (like that at the top of this post) and disregard the non-useful stuff (like the regional breakdown above). What’s more, they need to know what’s news and what’s non-news too, so they can make an informed decision on what to pitch as the former and what to advise their clients to pass on promoting.

Are you comfortable reading between the lines when it comes to statistics? If not, it might be time to brush up.

Posted in: statistics

Tags:

3Comments

Correlation/Causality Confusion

10.07.05

As the social media space slowly matures, we’re starting to see more and more reports released by companies offering insight into the ebbs and flows of peoples’ behaviour online. Not surprisingly, these reports are pretty popular – they possess two desirable attributes:

  • They offer easy-to-digest, soundbite-sized nuggets of information – perfect for short-form media such as Twitter
  • They offer the potential for increased understanding of the latest trends

However, these reports can sometimes unintentionally misrepresent the data, through a simple statistical error:

Correlation does not dictate causality.

What does this mean?

It means that just because there is a link between two things, it does not mean that one causes the other. So, if there is a correlation between A and B:

  • A could cause B
  • B could cause A
  • C (and/or D, E, F etc) could affect both A and B

As Bob Hoffman explains it over at The Ad Contrarian:

If you were to study people who are hard-of-hearing you would probably also find that they have a much higher likelihood of being bald. Does this mean that bad hearing causes baldness? Of course not. It occurs because old age causes both hearing and hair loss. So there is a correlation between deafness and baldness, but there is nocausality. One does not cause the other.

One more time: correlation does not mean causality. Just a quick thought, but an important one. Remember this next time you read a snippet about a new study, and make sure you always read closer before believing the headlines.

(Image: Shutterstock)

Posted in: statistics

Tags:

12Comments

Enough With Misusing Social Media ROI, Already

09.07.28

ROI-graph

I’m a little tired of abusing the term “ROI” – giving it new meanings just so they can say they’re measuring it. “Return on Interaction”… “Return on Engagement”… enough already.

Breaking news: ROI may well not matter for your social media program. (Edit: At least, not as a direct, immediate metric.)

Except this isn’t breaking news – people just don’t seem to hear it.

Here’s a definition of ROI from Wikipedia:

“Return on investment (ROI) [...] is the ratio of money gained or lost (whether realized or unrealized) on an investment relative to the amount of money invested.”

There’s even a formula:

ROI

ROI is a finanical term. It has a set definition, which carries plenty of weight in companies. However, that doesn’t mean you can always relate your programs directly to it.

For the formula to work, you need to know the cost and benefits of your program in dollar amounts. You should know the cost of your investment, but the gain may be hard to attribute (especially to a single factor). What’s the gain from improved customer service? From relationship-building? From increased employee engagement?

Sometimes you CAN identify a specific gain from your investment. Sometimes you can tie specific activity to conversions and have a specific value for those conversions. In those cases, you’re in luck – you’ve hit the communicator’s nirvana. The rest of the time, just accept it:

ROI may not be the right measurement for you.

Does that mean your program isn’t valuable? Does that mean you’ll never get executive sign-off? Does that mean it’s not worth measuring your program?

No.

It means you find appropriate ways to tie measurement back to your objectives. Those last four words are key: “back to your objectives.” Because everything should lead back to them.

As we’ve navigated through this recession, we’ve seen clients become (rightly) more and more focused on measuring outcomes, not outputs. It’s music to my ears, because this gives us the opportunity to (a) measure the heck out of a program and (b) adjust programs to ensure they achieve the right results for the client.

Those measurements don’t have to lead to a financial formula; they just have to tie back to your client’s goals. Do they want to drive sales? Address customer issues? Be perceived as leaders in their market? I could go on and on. Each of these has different end metrics, along with different proxies along the way. They’re all valuable.

So, please – enough with “return on influence” and other variations on the term “ROI.”

The fact that you’re not measuring ROI doesn’t mean you’re not measuring success or impact. In fact, it may just mean you’re measuring the right thing.

What do you think?

Update: Oliver Blanchard made an excellent point that I neglected to include here – Ultimately, all of these measures SHOULD feed back to ROI. If your company isn’t tying its activities back to that eventually, you risk both the cost of an ineffective program and the opportunity cost of missing more effective investment elsewhere. I would add that there may be intermediate steps between your program and the ROI calculation. Making-up new metrics because you can’t tie directly to ROI does nothing to help you.

(Images: Investopedia, Shutterstock)

Posted in: measurement, pr, public relations, social media

Tags: , , , , , ,

79Comments

Think PR People Don’t Need Math? Think Again

09.07.06

Public relations folks aren’t generally very good at math, according to their reputation, anyway. They’re creative people, you see? They work magic with words; they build relationships with people; they persuade people. So what if they’re no good with numbers?

I argue: it matters. A lot.

Here are just a few of the tasks that you need basic math to accomplish:

  • Social media and traditional media audits
  • MRP analysis
  • Social/traditional media monitoring analysis
  • Situational analysis for plans
  • Any kind of statistical analysis for data-based news releases
  • Market research analysis/recommendations

Get my point? “I’m no good with numbers” just doesn’t cut it.

I’ll admit it – I’m a math nut. I’ve worked as a data analyst and I nearly took a math degree (deciding instead to shoot for business) so I have a bit of an affinity for this stuff. Regardless, if you work in communications and you break into a sweat at the first sight of a graph, you need to study-up FAST if you want to progress.

No, just being able to create a graph in Excel does not cut it. Data alone does nothing. You need to be able to analyze that data. It’s not rocket science, but at a minimum you really should be able to, for example:

  • Compare two sets of numbers and calculate the percentage difference between the two
  • Know that 100% growth is different to 100% of something
  • Conduct simple statistical analyses of data – is there a trend?

In reality, we as a profession need to raise our game beyond statistical basics. We can’t just think about the numbers after the fact – a results focus needs to feature in every aspect of our work; especially online where data is so readily available. It takes planning and forethought to cut through the mass of data and turn it into useful, actionable, relevant information.

For a great example of how we should be building analysis points into all of our campaigns, check out this excellent post on integrating bit.ly and Google Analytics in a campaign. It shows a relatively simple process for integrating basic analytics into the links we publish on different social media platforms.

Results matter. That means numbers matter, and you need to know how to handle them.

Are you up-to-speed on these skills? How have you found ways to integrate analysis throughout your plans?

(Update: Radian6 just announced new features on their platform that may make this process easier – check out the Radian6 post on web analytics integration)

Posted in: pr, public relations, statistics

Tags: , , , ,

64Comments

New Research Provides A Social Media Reality Check

09.04.07

CNW Group and Leger Marketing today announced the results of new research into social media use in Canada (disclosure: CNW Group is a client).

The research provides a useful insight into social media trends along with some of the challenges that social media faces, but also sheds an interesting light on the differing perspectives between practitioners and regular social media users.

The top-level results are available online now. The full results will be released in a webinar on April 29 (register through the site).

Some key findings:

  • 49 per cent of social media users use social media at least once per day
  • 31 per cent of users agree that social media is more credible than advertising
  • 61 per cent are researching products to purchase
  • 36 per cent depend on social media to help them with purchase decisions
  • 40 per cent are “talking” to or learning from specific organizations
  • About one-quarter of users feel better about an organization that is engaged in social media
  • 89 per cent of users say they use social media the same or more than they did last year.

Once you dig down into these top-level facts, though, it gets more interesting.

User/Practitioner Gap

Social media is highly influenced by practitioners. For example, 19 per cent of social media users say their opinions are influenced by social media outlets, while 53 per cent of practitioners said the same – a significant difference. Similar, though smaller, differences show through in responses to other questions.

The implication of this is that practitioners often think that other people find social media to be more credible than they do in reality. 

There’s a gap between social media practitioners’ perceptions and those of users. However, given the time that social media has been around, the proportion that are influenced by social media is a good start.

Measurement is uncommon

Practitioners are generally only using broad objectives – there is a lot of room for improvement.

As well, few practitioners using social media tools are measuring what they do, and even fewer are going beyond looking at traffic. Interestingly, few managers are asking for this at this point. You can differentiate yourself by proactively digging deeper.

Room for improvement

While practitioners have a higher awareness of social media and its uses, they still think they, and organizations could use it better.

  • Few practitioners have a dedicated budget for social media
  • Few are monitoring social media (which astonishes me – I see it as a foundational piece for social media engagement)
  • Few practitioners are using social media for community building – most use it for marketing (although the lines blur in my eyes

Interesting stuff.

Which points stand out for you?

Posted in: social media, trends

Tags: , , ,

18Comments