![Measurement 101 A Presentation to the IPR Measurement Summit October 2009 Katie Delahaye Paine Member, IPR Measurement Commission www.instituteforpr.com CEO [email_address] www.kdpaine.com](https://image.slidesharecdn.com/2009measurement101-091015053923-phpapp01/85/2009-Measurement-101-1-320.jpg)
2009 Measurement 101
- 1. Measurement 101 A Presentation to the IPR Measurement Summit October 2009 Katie Delahaye Paine Member, IPR Measurement Commission www.instituteforpr.com CEO [email_address] www.kdpaine.com
- 2. Why Measure? “ The main reason to measure objectives is not so much to reward or punish individual communications manager for success or failure as it is to learn from the research whether a program should be continued as is, revised, or dropped in favor of another approach ” James E. Grunig, Professor Emeritus, University of Maryland “ If we can put a man in orbit, why can’t we determine the effectiveness of our communications? The reason is simple and perhaps, therefore, a little old-fashioned: people, human beings with a wide range of choice. Unpredictable, cantankerous, capricious, motivated by innumerable conflicting interests, and conflicting desires.” Ralph Delahaye Paine, Publisher, Fortune Magazine , 1960 speech to the Ad Club of St. Louis
- 3. The Ah-Ha moment, the Lotus PR Report Percent of impressions containing messages by product
- 4. The Ah-Ha moment, the Lotus PR Report Cost per message communicated
- 5. Victory Over Martians Confirmed! Share of exposure over time 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Jan Mar May Jul Sep Nov Jan Mar May Jul Sep Nov Jan IN TI MO NS
- 6. Interviews and media advisories generated best coverage 0% 20% 40% 60% 80% 100% Application articles Contract wins Exec Interview Media advisory Release + conference Press release plus VNR Product review Industry issue Trade show/event No Message Negative Message Positive Message
- 7. Comparing the effectiveness of different tactics
- 8. Correlation exists between traffic to the ASPCA web site and the organization’s overall media exposure
- 9. Correlations also exist between online donations to the ASPCA and the organization’s overall media exposure
- 10. Facebook: Correlating MSM, CGM and signups Strong correlation Non-negative discussion only
- 12. 10 signs that it’s the end of measurement as we know it BMC Software measures communications effectiveness based on contribution to EPS Procter & Gamble is now paying for engagement, not eyeballs Sodexo measures success of its Twitter program by showing $300K reduction in recruitment costs USO compares PR and PSAs to determine effectiveness of fund raising HSUS measures Social media success by calculating $650,000 in new donations Immunize BC counts calculates awareness + shots delivered The Red Cross measures the effectiveness of Twitter via lives saved and harm avoided IBM can calculate more sales from a $500 podcast than a $40K ad program BestBuy measures 85% lower turnover as a result of its Blue Shirt community Advertisers are starting to admit that all their measures are flawed
- 13. What do you need to measure to get to the Holy Grail of ROI measurement? Data. Outputs? Did you get the volume of coverage & visibility you wanted? Did you produce the promised materials on time and on budget? Outtakes? Did your target audience see the messages? Did they believe the messages? Outcomes? Did audience behavior change? Did the right people show up? Did your relationship change? Did sales increase?
- 14. Goals, Actions and Metrics Goal Action Output Metric Outtake Metric Outcome Metric Increased on-line donations Revamp website Amount of content on web site % perceiving ASPCA as a reliable source % increase in web traffi and donations #1 most trusted source for information on companion animals Increase staffing and resources for communications Increased exposure of “trusted source” message Increased perception of ASPCA as trustworthy and comprehensive source % increase in agreement with the statement Website is preferred site for information Add content, features to web site, keep up to date % increase in traffic % agreeing with the statement # 1 rankings, and time spent on site Integration of department Reorganize department More integrated materials produced on time and on budget Internal perception is “one department” Consistent messaging throughout
- 15. The 7 steps to PR ROI Define the “R” – Define the expected results? Define the “I” -- What’s the investment? Understand your audiences and what motivates them Define the metrics (what you want to become) Determine what you are benchmarking against Pick a tool and undertake research Analyze results and glean insight, take action, measure again
- 16. Step 1: Define the “R” what return do you expect? s What were you hired to change? If you are celebrating complete 100% success a year from now, what is different about the organization? If you eliminated your department or failed utterly, what would be different?
- 17. Generates Revenue, Sales, Profit Marketing Public Relations drives sales Investor Public Relations drives investment Public Relations drives donations & membership for relevant organizations Drives Efficiency Better audience targeting Reaching more people with a credible message for less money Avoids Catastrophic Cost Quality counsel helps to mitigate impacts of crises How PR Impacts Financial Performance
- 18. Increases likelihood to purchase / consider your brand(s) Minimizes the effects of a crisis Reinforces communication of organizational values Rebuilds trust after a crisis Establishes credibility of new products / companies; ease of market entry Commanding higher prices, lower costs, premium on stock price Enhances recommendations / word of mouth leading to faster adoption Increases customer loyalty / renewals / satisfaction Improves the attracting / retaining of talent Lowers legal costs How PR Positively Affects Reputation / Brand Equity
- 19. Increases employee satisfaction and engagement, leading to greater efficiency, increased retention, lower turnover rates, lower recruitment costs, and higher productivity Lowers legal costs Change employee behaviors such as greater levels of focus on key areas such as safety, quality, call response times Provides greater transparency and commitment to and from employees Creates a platform should it be necessary to communicate bad news at some stage in the future * Note items here can also refer to other internal publics such as trade association members . How PR Impacts Employees
- 20. Creates public awareness, understanding, and support for legislation, regulation, and political candidates Affects voter behavior Helps pass legislation, regulation, and initiatives Affects specific companies and industries through appropriations, tax impacts, and regulatory changes that can affect any and all aspects of a business Instigates and perpetuates grassroots or grasstops campaigns How PR Affects Public Policy
- 21. Step 2: Define your investment You can’t divide by $0 People time Opportunity Cost Executive time/goodwill
- 22. Step 3: Understand what motivates your stakeholders and priorities the stakeholders What motivates customers to purchase, members to join, students to apply, etc. What is your role in the outcome? Prioritize based on benefits
- 23. Step 4: Define your Key Performance Indicators (KPIs) The Perfect KPI Gets you where you want to go (achieves corporate goals) Is actionable Continuously improves your processes Is there when you need it
- 24. Step 4: Define your Key Performance Indicators (KPIs) carefully because you become what you measure Cost savings Efficiency Cost per message communicated Cost per new lead/customer acquired Productivity: Increase in employee engagement/morale Lower turnover/recruitment costs Engagement: Ratio of posts to comments % of repeat visitors % of 5+min visitors % of registrations Trust: Improvement in relationship /reputation scores with customers and communities (Loyalty/Retention) Thought leadership: Share of quotes Share of opportunities Message penetration Positioning on key issues Improvement in favorable/unfavorable ratio Improvement in Optimal Content Score (OCS)
- 25. Step 4: Potential KPIs for Media Relations Cost Per Message Communicated Increase in % of discussion containing key messages % increase in Optimal Content Score Share of discussion vs. the competition Share of brand visibility vs. the competition Share of spokesperson visibility vs. the competition Share of desirable coverage (positive+neutral coverage) vs. the competition Share of negatives Share of favorable positioning on key topics/battles Share of mentions by key media (including A-list bloggers)
- 26. Potential KPIs for thought leader visibility Our share of quotes in mass media Our share of papers in scientific/academic/trade journals Our leaders appointed to key professional boards over time Our share of papers at key conferences % awareness among “influentials” Total opportunities to see our message via speaking engagements
- 27. Potential KPIs for Community Relations Ratio between positive and negative press in local media % of articles in local media that contain our key messages % improvement in relationships scores between us and local community and those that influence the local community.
- 28. Step 5: Define your benchmarks Existing benchmarks Past Performance Peer companies Whatever keeps the C-suite up at night Think 3-5: A stretch goal The underdog who’s nipping at your heels Peer organizations Anyone that you compete with for share of mind or share of wallet
- 29. Past performance: tonality of blog content
- 31. Non-Profit industry benchmarks in social media % of Desirable Coverage 96% % of Key Message Communication 37% Number of Messages tracked 11.5 Words per Key Message 14 Number of Key Messages tracked 11.5 Most frequent conversation types Express support (69%) Making an observation (28%) % rallying support 3% % asking a question 4% % of exclusive mentions 17% % mentioning brand in title 8% % of discussions in Blogs 51% % of discussion on Twitter 30% % of visibility from YouTube & Flickr 10%
- 32. Overview of Key Metrics for Higher Ed Bookmark. Facebook Ext. Blogs Inst. Blogs YouTube MSM SOV 2% — 8% 9% 11% 7% Popularity 230 bkmks 500/mo. — 20 links 150k views — Engagement 59 cmts 1 day 13 cmts 2-12 cmts 2 cmts — % Positive 20% 32% 54% 50% 15% 15% % Negative 0% 0% 4% 0% 1% 2% Strat. Mess. 40% † 18% † 42% 42% † 18% 38%
- 33. Step 6: Conduct research (if necessary) First: find out what already exists Web traffic Customer Satisfaction data Customer Loyalty data Second: Decide what research is needed to give you the information you need:
- 34. Step 6: Selecting a measurement tool Objective KPI Tool Increase inquiries, web traffic, recruitment % increase in traffic #s of clickthrus or downloads Omniture, Google Analytics, Web Trends Increase awareness/preference % of audience preferring your brand to the competition SurveyMonkey, Zoomerang Engage marketplace Conversation index greater than .8 Rankings TypePad, Technorati , Radian6, Communicate messages % of articles containing key messages Total opportunities to see key messages Cost per opportunity to see key messages Media content analysis – Dashboards % aware of or believing in key message Survey
- 35. Your tool box needs: A content source: Google News/Google Blogs, RSS feeds Technorati, Social Mention, Twazzup, Cyberalert, CustomScoop, e-Watch Radian 6, Techrigy, Sysymos, Visible Technologies, Scout Labs Survey Monkey/Zoomerang
- 36. Your tool box also needs to include: 2. A way to analyze that content Automated vs. Manual Census vs random sample The 80/20 rule – Measure what matters because 20% of the content influences 80% of the decisions Dashboards to aggregate data
- 37. The Content Analysis Process Training Are we all on the same page? Content Collection Did we get everything? Coding Start analyzing (computers/humans) Final Reliability (Resource Permitting) Reporting What does it all mean? Adapted with edits from The Content Analysis Guidebook (Neuendorf , 2002) .
- 38. Why an Optimal Content Score? You decide what’s important: Benchmark against peers and/or competitors Track activities against OCS over time
- 39. Optimal content score for media coverage Positive: Mentions of the brand Positive brand mentions Key messages Customer quoted positive Analyst quoted Positioned as trusted partner Positioned as trustworthy, ethical Positioned as vendor of choice Positioned as global Positioned as one company Negative Omitted Negative tone No key message No quote No analyst quote
- 40. How to calculate Optimal Content
- 41. Charting OCS over time between divisions
- 42. Trend against competition with OCS
- 43. Standard classifications of discussion Acknowledging receipt of information Advertising something Answering a question Asking a question Augmenting a previous post Calling for action Disclosing personal information Distributing media Expressing agreement Expressing criticism Expressing support Expressing surprise Giving a heads up Responding to criticism Giving a shout-out Making a joke Making a suggestion Making an observation Offering a greeting Offering an opinion Putting out a wanted ad Rallying support Recruiting people Showing dismay Soliciting comments Soliciting help Starting a poll Validating a position
- 44. Standard classifications of videos Advertisement Animation Demonstration Event/Performance Fiction Film Home Video Instructional Video Interview Lecture Montage Music Video News Broadcast Promotional Video Sightseeing/Tour Slideshow Speech Television Show Video Log
- 45. Your tool box also needs to include: 3. A way to measure engagement The conversation index Links/trackbacks Time spent Relationship studies Questionnaires
- 46. Share of conversation vs share of engagement Page 2 2 1 2 1 6 5 3 1 1 1 1 1 1 1 1 1 1 2 2 2 1 2 1 2 1 4 2 1 4 2 1 1 4 1 6 7 6 2 2 2 2 1 3 2 3 1 0 2 4 6 8 10 12 14 16 18 20 Faculty Students Research, Physical Sciences Courses Research, Earth Sciences Projects, Non - Research Financials Alumni Topics Research, Life Sciences Staff Admissions Legal News Other Research, Agriculture Policies Institution, Overall Campus Life Research, Social Sciences Share of Subject Peer 1 Michigan State Peer 2 Peer 3 Peer 4 15.3% 68.7% 100.0% 4.4% 33.3% 96.8% 28.6% 34.9% 12.5% 43.3% 28.6% 13.0% 38.3% 100.0% 23.6% 66.7% 6.3% 28.6% 20.8% 2.3% 95.6% 33.2% 5.8% 28.6% 100.0% 86.8% 13.0% 31.0% 22.1% 3.2% 71.4% 43.5% 18.8% 94.2% 56.7% 14.2% 13.2% 53.2% 28.4% 21.1% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Admissions Alumni Topics Campus Life Community Relations Courses Events Faculty Financials Institution, Overall Inventions Legal News Other Partnerships Policies Projects, Non - Research Research, Agriculture Research, Earth Sciences Research, Life Sciences Research, Other Research, Physical Sciences Research, Social Sciences Staff Students Share of Engagement by Subject - ,External Blogs Peer 1 Michigan State Peer 2 Peer 3 Peer 4
- 47. The vast majority of discussion in external blogs is neutral. Page
- 48. One definition of engagement % increase or decrease in unique visits In the past month, what % of all sessions represent more than 5 page views % of sessions that are greater than 5 minutes in duration % of visitors that come back for more than 5 sessions % of sessions that arrive at your site from a Google search, or a direct link from your web site or other site that is related to your brand % of visitors that become a subscriber % of visitors that download something from the site % of visitors that provide an email address Courtesy of Eric Peterson
- 49. Your tool box also needs to include: 5. Measuring ROI Membership Registrations WebSide story/Web trends/Clicktraks Sales tracking Marketing Mix Modeling
- 50. Research without insight is just trivia What works, what doesn’t What needs to be done? What are you communicating? What tools work best? Step 7: Analysis
- 51. Data mining the numbers you have Look for failures first Then look for exceptional success Compare to last month, last quarter, last year Figure out what worked and what didn’t work
- 52. Actionable Conclusions A sk for money Get C ommitment Manage T iming I nfluence decisions Get O utside help Just Say N o
- 53. Measuring doesn’t have to be complex High Impact Low Cost High Costs Low Impact Blogging Intentional Leaks SEO-optimized press releases e-newsletter Advertising Press conference Party
- 54. Take action and measure again Make sure data is ready when you need it Work around regular reporting schedules Keep questions and criteria consistent Benchmark Conclusions Report
- 55. Mistakes made most often in the measurement process Lack of agreed upon goals Questionable methodology Unclear definition of tone/sentiment Incorrect search strings Incomplete or out-of-date media list (no CGM) Incorrect circulation figures New products/spokespeople not added to analysis parameters Promising a Jaguar on a bicycle budget Not allowing enough time to do measurement right Insufficient so what – looking at the trees not the forest
- 56. Thank You! For more information on measurement, read my blog: http://kdpaine.blogs.com or give me your card and we’ll send you The Measurement Standard, www.themeasurementstandard.com To start developing your own dashboard or for a copy of this presentation go to: http:// www.kdpaine.com Or call me at 1-603-868-1550 Follow me on Twitter: @kdpaine Friend me on Facebook: Katie Paine Skype: KDPaine
Editor's Notes
- #6: This was perhaps the moment when I realized I had the ultimate Silicon Valley dictionary. I was presenting results to a roomfull of pocket-protector wearing engineers, all highly annoyed that they’d been dragged away from what was obviously a far more important conversation about that cool invention over on the next bench. Our assignment had been to look at how much total coverage the leading publications gave to the subject of microcontrollers and to determine if they were getting their fair share. We actually studied the magazines over a three year period and discovered that for first 24 months we studied, the four leading vendors essential most of the 36 months
- #18: Here we see how PR affects financial performance whether it be sales of a company’s products, or donations to a non profit. But, PR can also make a company’s marketing more efficient through better audience targeting and less costly approaches to reach an audience. Also, PR can help an organization weather a crisis and avoid the catastrophic loss, for example, of a destroyed corporate reputation by not effectively telling one’s side of the story.
- #19: A huge role for PR is improving reputation, image, or brand equity. The results of doing so are listed here.
- #20: While we talk about employees in these next two charts we are referring to a broader set of internal publics such as contractors, business partners, and members of an organization. Basically, it is about a stronger bond between an organization and its internal publics coming from the skilled PR practitioner’s efforts. And those bonds create better business results.
- #21: PR affects public policy usually in the form of public affairs activities. The result is a change in a realm of political, regulatory, or legislative outcomes. From a company’s perspective, it is how those changes affect the bottom line.