SharePoint MoneyBall: The Art of Winning the SharePoint Metrics Game by Susan Hanely - SPTechCon
4 likes1,061 views
This document discusses metrics for measuring the success of a SharePoint implementation. It emphasizes establishing business objectives and stakeholder needs before selecting metrics. Both quantitative metrics like time savings and qualitative metrics like user stories are recommended to capture different types of impacts. The metrics should be presented to stakeholders in a way that is relevant and collected at low cost.
![Sample Business Metrics
Hours per week to execute a process
Number of Proposals/Contracts per year
Number of “[My Organization]-All” emails
Number of email attachments
Average application training costs
Cost savings to retire a legacy application
22](https://image.slidesharecdn.com/hanleysharepointmoneyball-130212155955-phpapp01/85/SharePoint-MoneyBall-The-Art-of-Winning-the-SharePoint-Metrics-Game-by-Susan-Hanely-SPTechCon-22-320.jpg)
![Survey Example: Post-Pilot Usability
Usability Question Metric
If presented the choice, do you want to keep the “Don’t Take it
solution? Away”
Don't take it away Take it away
Usability/friendliness - how does the usability of this “User Friendliness
solution compare to other solutions you use on a Rating”
regular basis?
Much easier to use Harder to use
Easier to use Much harder to use
About the same
How easy and intuitive was the solution to use for each “Intuitiveness
of the following [specific task]? Rating”
Very easy Difficult
Easy Very Difficult
Moderate
49](https://image.slidesharecdn.com/hanleysharepointmoneyball-130212155955-phpapp01/85/SharePoint-MoneyBall-The-Art-of-Winning-the-SharePoint-Metrics-Game-by-Susan-Hanely-SPTechCon-49-320.jpg)
SharePoint MoneyBall: The Art of Winning the SharePoint Metrics Game by Susan Hanely - SPTechCon
- 1. SharePoint Moneyball – The Art of Winning the SharePoint Metrics Game March 4, 2013 ©2013 SUSAN HANLEY LLC
- 2. About Me • President, Susan Hanley LLC • Led national Portals, Management Collaboration, and Content practice for Dell • Director of Knowledge Management at American Management Systems • Governance • User Adoption • Metrics • Information Architecture • Knowledge Management • Portals • Collaboration Solutions [email protected] susanhanley
- 3. Why Measure? – The Four “F” Words Feedback Funding Follow-on Focus 3
- 4. Measurement throughout the life-cycle Before During After Make the business Provide a target Develop case benchmarks Make tradeoffs Develop lessons Tune the learned implementation process 4
- 5. Make your case for the solution SharePoint enables 5
- 6. Measurement Process 4. What are the 1. What are the Modify the METRICS and how BUSINESS OBJECTIVES? measures should we PRESENT them? 2. How should the SOLUTION be Modify the 5. How can we COLLECT DESIGNED to meet the process or tool the metrics? objectives? 6. What do the metrics 3. Who are the metrics Aid decision TELL US about how we STAKEHOLDERS? making need to CHANGE? 6
- 7. 1. What are the BUSINESS OBJECTIVES? Without a critical business initiative … …“Career limiting move” 7
- 8. Be the main event
- 9. It’s easy to go for the “motherhood” objectives … More innovative products and services More effective marketing Better access to knowledge Lower cost of doing business – reduction in travel and other operational costs Higher revenues Improved employee, customer, and partner satisfaction 9
- 10. It’s better to Get SMART! Specific (concrete and well-defined) Measurable (quantifiable, comparable) Achievable (feasible, actionable) Realistic (consider resources) Time-bound (deadline driven) 10
- 11. SMART objective for a proposal library Reduce the average amount of time it takes to produce complex Reduce the average amount oftime it takes to to produce time it takes produce complex Reduce the average amount ofoftime it takes to produce average amount proposals by 10% in 10% in the complex proposals the next yearnext year proposals by 10% by next year complexproposalsin by 10% in the next year Specific Measurable Achievable Realistic Time-bound 11
- 12. Considering SharePoint 2013? Search Connecting Mobile Cloud Support People 12
- 13. 2. How should the solution be DESIGNED to meet these objectives? Site Technical Features Architecture Infrastructure Customization Security Governance Roles and Responsibilities Training and Communications 13
- 14. Your business case is personal
- 15. 3. Who are the metrics STAKEHOLDERS? They’re at all levels - especially in the middle They care about different things 15
- 16. For each key stakeholder, ask … What counts? What keeps you up at night? What do you already use? What do I need to tell you? 16
- 17. 4. What are the METRICS and how should we PRESENT them? Identify the type • Quantitative Consider the life-cycle • Qualitative Establish a baseline Gain commitment Decide the best way to about targets communicate 17
- 18. Good metrics come in multiple types … plan on both Quantitative Performance between points Spot trends Qualitative Provide context Used when numbers aren’t easy (storytelling) Used at early project stages (future scenarios) Richer (“serious anecdotes”) 18
- 19. Return on Investment Benefit > Cost Be careful: whoever controls the spreadsheet and the assumptions can make an ROI that can justify anything. Resources: Total Economic Impact™ of Microsoft SharePoint Server 2010 (Forrester) http://bit.ly/cWfeyN Best for platform investment, less helpful for individual solutions Cloud vs. On Premise Calculator (Andrew McAfee and Google Analytics) http://bit.ly/R6jlsZ (for small to medium businesses) 19
- 20. ROI is only PART of the story Good metrics are: Related to outcome Relevant to stakeholders Collected at low cost Balanced 20
- 21. Consider two other types of quantitative metrics BUSINESS METRICS SYSTEM METRICS 21
- 22. Sample Business Metrics Hours per week to execute a process Number of Proposals/Contracts per year Number of “[My Organization]-All” emails Number of email attachments Average application training costs Cost savings to retire a legacy application 22
- 23. Business Metrics Example: Best Practice Library • Save time, improve consistency by not re- Business Goal inventing the wheel • SMARTer Objective: Reduce the number of employee reported hours per week searching for examples of prior work. Approach System Metrics • OK: Number of views per document • Better: Ratings Business Metrics • Best: Combine both with survey to seek out specific re-use cases 23
- 24. Metrics Example: Prioritizing Resources for Projects Allocate limited SharePoint Resources for Business Goal • Process Improvement Projects • SMART Objective Example – Reduce the amount of time for a task by x% in 90 days Approach x x x T = time on task (in minutes) E = number of employees performing that task N = number of times per year a typical employee performs that task S = average employee’s loaded salary per minute 24
- 25. Qualitative Metrics – the stories that drive it home Keep it real In the storyteller’s words Serious Anecdotes 25
- 26. Serious Anecdote | Consulting I joined the organization on March 16 without previous experience. After one week of training, I joined a project team. After one day of training on the project, I was assigned a task to learn a particular technology that was new to everyone on the team. I was given a bunch of books and three days, my tasklearn how to Instead of told that I had three days to was create a project using this technology. completed in 4 hours. In my first week of training, I learned about the company’s intranet where people described their expertise. I sent an email to four people I found with a search for that technology asking for their help. One of them sent me a link to a document containing exactly what I needed. 26
- 27. Serious Anecdote | Pharma – The Need A scientist with Thrombotic & Joint Diseases in Germany began a project to isolate and culture macrophages and needed some help. Meanwhile, two scientists in the US had deep experience in protocols for this area. 27
- 28. Serious Anecdote | Pharma – The Result The German scientist consulted the expertise directory to find that expertise existed within the company and contacted the two US scientists he found in his search. Both scientists quickly responded with assistance. One Benefit: The German scientist was able to helped him with culturing leverage existing internal expertise and, protocols and the other helped in the process, reduce his research effort him with information on magnetic cell sorting. by four weeks. 28
- 29. Different Measures for Different Stages Enterprise Growth Phase Use mixture of metrics to show value across the organization. Pilot Project Phase Solution Maturity Use definitive metrics to show real value for business objectives. Startup Phase Use scenarios and simulations to explore projected results and effects. Pre-Planning Phase Use scenarios and simulations to explore projected results and effects. Time 29
- 30. How do you spell success? Have a Baseline and Target 30
- 31. Presenting Metrics Balanced Scorecard Dashboard “Report Card” 31
- 32. Balanced Scorecard Dimensions Business Value Health Capabilities 32
- 33. Balanced Scorecard Example | Expertise Location Metric Target Pilot Outcome # of Anecdotes 10 serious 10 serious + 12 transactional Business Minimum $ value/anecdote $X $2X Value: Estimated time saved X months X + 3 months # searches/user/week .25 .58 Usefulness rating 3.5 out of 5 3.6 out of 5 Health: % of users who say “Don’t 66% 83% take it away” Usability/friendliness rating 3.5 out of 5 4.1 out of 5 # Anecdotes (repeat metric) 10 serious 22 Capabilities & Culture: % of participants attending 75% 85% training 33
- 34. 5. How can we collect the metrics? Try not to over-achieve – balance counting with “doing” Automate where possible Get creative when it comes to qualitative metrics Ask Survey Usability Testing Active Listening Seek Send out a “journalist” Track Classify by type Keep storyteller value metrics – what was the benefit to you? 34
- 35. Example Survey Questions If given the choice, would you KEEP it? How does this COMPARE? How EASY was it to …? 35
- 36. Tip: Collect Qualitative Metrics with Ratings Was it helpful? Were you able to get value? Trigger a Can we call Follow up survey at 4-5 you? conversation 36
- 37. 6. What do the metrics tell us about how we need to change? Are we doing the right thing? What areas are most successful? What areas should we be promoting? In which areas should we be investing? Which initiatives should we discontinue? 37
- 38. Keep in mind Metrics alone won’t make your program successful A person who is A person whose job it accountable for making is to monitor them changes based on analysis It’s as important to have a plan for acting on metrics as it is to have a plan for collecting them! 38
- 39. Call to Action Make sure metrics Develop a plan to capture quantitative are part of and qualitative metrics. someone’s job. Identify baseline measures – and gain commitment on targets – before you start!! Develop a library or list to Develop an approach to capture and categorize produce and promote metrics. qualitative metrics. 39
- 40. Susan Hanley Susan Hanley LLC [email protected] 301 469 0770 (o) 301 442 0127 (m) @susanhanley www.susanhanley.com http://www.networkworld.com/ community/sharepoint 40
- 41. Extras 41
- 42. White Paper For a white paper that explains the concepts in this presentation in more detail – with lots more examples, please go to http://www.susanhanley.com. 42
- 43. Examples of Intranet System Measures Key Measure Objective Metric Which features of the Knowing which pages are most used can help to prioritize which pages should Page Hits intranet are most be improved or developed. “Dwell” Time (Time on important? You can also see which business units are the biggest intranet users and which Page/Site) business unit’s content is used the most. Which features are not If certain pages have low usage numbers, it is an indication that either the page Page Hits being used? is not very popular—and therefore should be a lower priority to develop—or Document Downloads that people are just not aware of its existence (which might be a communications or "promotion" problem). Is the site navigation A high number of hits on a page that is not easily accessible from the main page Page Hits on pages deep effective? indicate that the popular page should be moved up in the hierarchy. in the hierarchy Search results with no hits present opportunities to both promote content and add best bets. Which team sites Sites that have not been accessed in the past 12 months might be candidates for Page Hits should be archived or archival or deletion if the content is no longer useful. deleted? What are the Monitoring usage trends helps identify patterns or problems and potentially Usage by time peak/low usage alerts the Exchange Business Owner and Portal Administrator of potential user or times? performance issues – ideally, before they become a problem. How is usage Trending reports are available for a limited period of time within SharePoint Number of users and trending? 2010. Third-party tools are required to do multi-year detailed trend analysis. number of unique users over time 43
- 44. Sample System Metrics (“out of the box” SharePoint 2010) Metric Objective Number of Unique •Provides a proxy for adoption, which is a loose proxy for value. Users (month to month) Most Viewed •Provides a proxy for the most valuable content. Pages/Sites •Sites not being used help identify content that might either need to be promoted or deleted. Top Queries (search) •Identifies “trending “ content. •Top queries can also provide insights about what content should be promoted to the home page. Failed Queries / No •Identifies candidates for best bets or synonyms and identifies emerging Results Queries business terms or concepts. Best Bet Suggestion •Helps the business owner improve user outcomes by identifying URLs as Report most likely results for search queries. Best Bet Usage •Helps identify which best bets are adding value and as an input to determine new best bets or best bets that need updating. 44
- 45. Additional Useful System Metrics (third-party for 2010) Metric Objective Most Viewed Documents Provides a proxy for the most valuable content. Document Contribution/Editing Analysis Provides a way to measure sustained adoption from the perspective of employee engagement. Team Site Summary Information Provides a way to understand which sites are actively being Total Number of Team Sites used to monitor the health of the collaborative team sites. Viewed in Past 30 days Can be used to identify which sites are no longer being Modified in Past 30 days used and might be able to be deleted or archived. Sites with no access in past 12 Provides a proxy for whether or not team sites are adding months value. Trend of the number of team sites created My Site Summary and Trends Identifies adoption of people-to-people collaboration Total number of My Sites features. Viewed in past 30 days Proxy for employee engagement. Modified in past 30 days Average size 45
- 46. System Metrics in SharePoint 2013 Online 46
- 47. One sample metric for each stage (more in White Paper) Life-cycle Example Quantitative Metric Example Qualitative Metric Sources Stage Planning • Time to perform current • "Day in the life" future • Work measurement studies process stories • Interviews of key stakeholders Start up • N/A • Immediate term “day in the • Employee surveys life” stories Pilot • Same metrics you used for • Usage anecdotes –specific • Follow up work Conclusion baselines examples from pilot measurement studies • Surveys and follow up interviews Ongoing • Additional metrics relevant • Usage anecdotes with a • New solution system to the business problem “serious” punch line that metrics available with new process you collect and catalogue • Employee surveys and on an ongoing basis follow up interviews 47
- 48. Balanced Scorecard Framework Objectives Critical Success Factors Source Sample Metrics Gain frequent and • High volume of needs that can’t be • System metrics • # of searches per week sustained met through existing channels • User Surveys • # of average users per week Solution adoption of • Positive impact on existing workload • # unique users per week Health solution or work processes • # of “hits” on key pages/sites • “Usefulness rating” from user surveys • % of users who say “don’t take it away” at the end of the pilot Provide reliable, • Solution user-friendliness and • System metrics • “Usefulness rating” from user surveys easy-to-use intuitiveness • User Surveys • # of searches per week technology that • Solution reliability • Direct measurement • # of average users per week can be • Integration of the solution with work • # unique users per week incorporated into processes and existing tools • # of “hits” on key pages/sites work processes • # Help Desk calls/week Capabilities Ensure users • User training • System metrics • % of users trained understand • Effective help resources • User Surveys • % of pilot milestones achieved objectives and • Persistent, clear communications • Direct measurement • # of communications events/activities how to leverage • Active, sustained management the solution support • Incorporation of collaboration into performance objectives and evaluations Demonstrate clear • Tangible, quantifiable examples of • “Serious” Anecdotes • # of anecdotes Business value with respect reductions in process cycle time collected via surveys • $ value of anecdotes Value to the business • Estimates and/or • Cycle time improvement (in hours) strategy direct measurement of cycle time 48
- 49. Survey Example: Post-Pilot Usability Usability Question Metric If presented the choice, do you want to keep the “Don’t Take it solution? Away” Don't take it away Take it away Usability/friendliness - how does the usability of this “User Friendliness solution compare to other solutions you use on a Rating” regular basis? Much easier to use Harder to use Easier to use Much harder to use About the same How easy and intuitive was the solution to use for each “Intuitiveness of the following [specific task]? Rating” Very easy Difficult Easy Very Difficult Moderate 49
- 50. Other Resources How to Measure Anything: Finding the Value of Intangibles in Business by Douglas Hubbard Jakob Nielen’s Alert Box - Current Issues in Web Usability: http://www.useit.com/alertbox/ Determining the Value of Social Business ROI: Myths, Facts, and Potentially High Returns: http://www.idc.com/getdoc.jsp?containerId=225497 Social Software for Business Performance: The missing link in social software: Measureable business process performance: http://www.deloitte.com/assets/Dcom- UnitedStates/Local%20Assets/Documents/TMT_us_tmt/us_tmt_socialsoftwareexecsu mmary_021411.pdf SharePoint Lifecycle Management Solution with Project Server 2010: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=17058 50