Downloaded 11 times
Stephanie Faulkner - Moving Beyond Citation Indexes – Measuring Societal Impact through Clinical, Policy and Patent Citations
Plum Analytics, founded in 2012 and now part of Elsevier, emphasizes the importance of measuring societal impact through diverse metrics, including clinical, policy, and patent citations. The document highlights the challenges and necessity of demonstrating real-world impact of research, particularly in influencing policy and addressing societal issues. PlumX aims to enhance its metrics to better reflect the significance of clinical research and deepen the connection between academia and industry.
More Related Content
Similar to Stephanie Faulkner - Moving Beyond Citation Indexes – Measuring Societal Impact through Clinical, Policy and Patent Citations
More from SciELO - Scientific Electronic Library Online
Stephanie Faulkner - Moving Beyond Citation Indexes – Measuring Societal Impact through Clinical, Policy and Patent Citations
- 1. | www.plumanalytics.com1 Moving BeyondCitation Indexes – Measuring Societal Impact through Clinical, Policy and Patent Citations 26 September 2018 Stephanie Faulkner | Product Manager [email protected] | +1.732.216.5104
- 2. | www.plumanalytics.com2 Who isPlum Analytics? • Founded to Tell the Story of Research • Developed comprehensive metrics across 5 categories (Citations, Usage, Captures, Mentions, Social Media) • Timeline: − 2012: Company founded − 2014: Merged with EBSCO − 2017: Joined Elsevier • 250 worldwide customers, including: − University of Southern California − University of Pittsburgh − Universidad del Rosario − University of Western Australia − University of Helsinki, Finland − Chalmers University, Sweden − Nanyang Technical Institute, Singapore − UNICEF, Italy − CSIRO, Australia − University of Sydney, Australia
- 3.
- 4.
- 5.
- 6.
- 7.
- 8. | www.plumanalytics.com8 What areyour institution’s / country’s strengths vs. your peers What research areas are getting funding? How can we demonstrate ROI (Return On Investment) of research money? Who are rising stars amongst early career researchers? Use metrics to tell the story of research Research Metrics Can Be Used to Identify…
- 9. | www.plumanalytics.com9 • Fundersand governments increasingly require real world or societal proof – consider the 2014 Research Excellence Framework exercise conducted in the UK. • 6,975 impact case studies were submitted categorized by a variety of impact types including: − Political − Health − Technological − Economic − Legal − Cultural − Societal − Environmental • Almost 25% of all the cases submitted were labeled Societal Impact http://impact.ref.ac.uk/CaseStudies/ https://www.ugc.edu.hk/doc/eng/ugc/rae/2020/framework.pdf Societal Impact – Why does it matter? 9
- 10. | www.plumanalytics.com10 • Measuringsocietal impact is not always easy as it can take a long time for this type of impact to be reflected and measurable. • Sources vary and the accessibility to them also varies. • There is no standard measure across research areas. Societal Impact - Challenges 10
- 11. | www.plumanalytics.com11 Measuring SocietalImpact – PlumX Metrics
- 12. | www.plumanalytics.com12 • Werepresent societal/economic impact in our Citations area in three ways: − Clinical Citations − Policy Citations − Patent Citations • Displayed separate from traditional citation indexes, but included in our overall citation count Our approach 12
- 13. | www.plumanalytics.com13 Policy andClinical Citations • Demonstrates: − Public engagement with research − Impact on government policy and cultural life − Societal impact in general • Can drive policy makers and legislators to a particular course of action. • They can also be actual statements of policy by public policy making institutions.
- 14. | www.plumanalytics.com14 Policy Citations Policyfrom different types of organizations: • Non-profits • Governmental agencies • NGOs • Think tanks Policy documents can be: • Guidelines • White Papers • Monographs • Pamphlets • Articles • Books • and more
- 15. | www.plumanalytics.com15 Societal Impact– Policy Citations
- 16.
- 17.
- 18. | www.plumanalytics.com18 “…clinical medicinedraws heavily on basic science, but not vice versa. The result is that basic medicine is cited three to five times more than clinical medicine, and this is reflected in journal impact factors.” Clinical Research & Citations
- 19. | www.plumanalytics.com19 Clinical CitationsCome From… Clinical Alerting Services Clinical Guidelines Systematic Reviews Clinical Trials
- 20. | www.plumanalytics.com20 Clinical CitationSources: Practice Guidelines • Based on PubMed / Medline − “Guideline” is a classification by PubMed − Any time an article is cited by a “Guideline” this will be counted as a clinical citation − PlumX will calculate this as a type of “clinical citation count” for a given article Note we will also have a separate “citation index” count for citations across all of PubMed. • Based on NICE, UK (Clinical Guidelines) • And many more Guidelines to follow…
- 21.
- 22. | www.plumanalytics.com22 Societal Impact– Clinical Citations
- 23. | www.plumanalytics.com23 Clinical CitationSources: DynaMed Plus Topics • Clinical reference tool • Designed for point-of-care • Content: Extensive network of clinical experts − Review 500 journals and 120 guideline organizations − select the best and most appropriate evidence − confirm the clinical applicability of content − peer-review topics
- 24. | www.plumanalytics.com24 Societal Impact– Clinical Citations
- 25.
- 26.
- 27. | www.plumanalytics.com27 Patent Citations •A patent citation is any document cited by a patent applicant, a third-party, or a patent office examiner because it's relevant to a patent application. • Gain an understanding of how much research is being used in the creation of products by seeing the total number of outputs that received patents. • Provide information about the economic impact of research.
- 28. | www.plumanalytics.com28 Patent Citations •Citation of scholarly output in patents indicates a connection between academia and industry. • Patents citation contains valuable data and if analyzed well, may sometimes reveal concealed mysteries of the information flow between countries, laboratories, companies, and universities. Sharma, P., & Tripathi, R.C. (2017). Patent citation: A technique for measuring the knowledge flow of information and innovation. World Patent Information, 51, 31-42. doi: 10.1016/j.wpi.2017.11.002
- 29. | www.plumanalytics.com29 Societal Impact– HERL at University of Pittsburgh
- 30. | www.plumanalytics.com30 Societal Impact– Patent Citations
- 31. | www.plumanalytics.com31 Societal Impact– Patent Citations
- 32. | www.plumanalytics.com32 Summary • It’sexciting to see the real-life impact that research can take – whether impacting law, medical care or the invention of new technologies. • Clinical research is an area underserved by traditional citation metrics, yet is important to the day- to-day welfare of people around the world. Being able to measure the impact of clinical research can encourage more clinicians to innovate and find cures for diseases. • PlumX will continue to add new clinical, policy and patent sources and innovate around additional measures to help further demonstrate societal impact.
- 33.
- 34. Thank You [email protected] +1.732.216.5104 Stephanie Faulkner Findout more information at: http://www.plumanalytics.com
Editor's Notes
- #5 If you don’t know where to eat you will choose metrics such as restaurant reviews and ratings to choose a restaurant.
- #6 Businesses like Starbucks use metrics like foot traffic, auto traffic, costs, etc. to determine the best place to locate a new store.
- #7 Uber has used metrics to optimize revenue by implementing surge pricing when they detect a big demand.
- #8 Thus, comprehensive metrics give you an edge.
- #9 Research metrics can be used to identify An institution, journal, countries strengths What research areas are getting funding Return on investment (particularly for funders) What new researchers are getting recognition Metrics can help you tell the story about research
- #10 More and more regional assessment exercises – like the REF in the UK, the ERA in Australia and the RAE 2020 in Hong Kong, where they are specifically looking to assess the reach and significance of impact on the economy, society and/or culture -- need to have these types of societal impact metrics for these types of assessments. Funders are asking for metrics that demonstrate real world proof of the how a piece of research has made an impact. Researchers and Institutions want to know what impact their work has made. The purpose of the REF in the UK, for example, is to provide assessment outcomes for each submission from higher education institutions. These outcomes will be used for or by: The 4 higher education funding bodies to help determine the allocation of research grants to which institutions they fund Accountability for public investment in research and it’s benefits Benchmarking information and establishing reputational yardsticks, for use within the Higher Education sector and for public information. In addition, there are 3 other benefits identified: to provide a rich evidence base to inform strategic decisions about national research priorities to create a strong performance incentive for higher education institutions and individual researchers to inform decisions on resource allocation by individual higher education institutions and other bodies Just looking at the last REF in the UK from 2014, they received almost 7,000 impact case studies which the classified into a variety of categories. Of the almost 7,000, 25% of the cases were labeled as Societal impact. As the REF prepares for the 2021 report, they state that they are specifically assessing the ‘reach and significance’ of impacts on the economy, society, culture, public policy or services, health, the environment or quality of life – Societal Impact.
- #11 Measuring this type of impact can be a challenge – patent applications can take 18 months or longer (in the US) to be approved and have the reference data publicly available. Many governments do not have policy documents (and the associated references) in a format for automated harvesting, so manual curation is needed. And how do you equally measure the impact of an astrophysicist compared to an economist? It’s like comparing apples to oranges. Source: “'What is Real World Impact?’ Looking at Impact Across a Spectrum of Influence, Attention and Affecting Change ” presented at the Society for Scholarly Publishing 2018 annual conference. By David Crotty of the Oxford University Press.
- #13 PlumX currently represents societal and economic impact in our citations metric category, but separate from traditional citation indexes. We currently categories our societal and economic impact metrics as: clinical citations (which we have pioneered and I’ll speak more about in detail), policy citations and patent citations
- #14 Let’s first talk about Policy Citations. We know that finding references to research in policy documents is important because there are new demands on researchers and those who support them to demonstrate public engagement with their research, its impact on government policy and cultural life, and societal impact in general. These policy documents can be designed to drive policy makers and government officials to a particular course of action. They can also be actual statements of policy by public policy making institutions like a government.
- #15 However, the term “policy documents” can be a vague way of describing documents that result from the research of non-profits, governmental organizations and think-tanks. They can take the form of white papers, monographs, pamphlets, articles, books, book chapters or reports. They can contain words in the title like “Policy”, “Guidelines”, “Recommendations” or “Guidance” or they may just be research objects that are used in the creation of policy.
- #16 For example, the World Health Organization’s primary role is to direct and coordinate international health within the United Nations system. They work to combat diseases influenza and HIV, cancer and heart disease. They help mothers and children survive and thrive so they can look forward to a healthy old age. They ensure the safety of the air people breathe, the food they eat, the water they drink – and the medicines and vaccines they need. The World Health Report in 2013 focused on "Research for universal health coverage" argues that universal health coverage – with full access to high-quality services for prevention, treatment and financial risk protection – cannot be achieved without the evidence provided by scientific research. This article on the SciELO platform was cited in that WHO 2013 report on universal health coverage.
- #17 In another example, not from the SciELO platform, you can see that this article has received 5 different policy citations. Let’s take a look at the policy details…
- #18 If you look at the title and source of the policy documents, you can see that this one article had an impact on food and nutrition policy in different ways, including economic impact with both the World Health Organization and the World Bank referred to this article in their policy works.
- #19 With regards to clinical research, we know that it does not get cited as much as basic research---recognized as far back as 1997. Why is this a problem? Researchers have difficulty getting funding so early career researchers opt out of translational medicine. Hospitals and research institutions cannot showcase their talents. It hurts us all when our researchers don’t want to turn basic science into ways to treat illness and disease. Or, when our doctors pioneer new treatments but have no motivation to publish them. We need to measure clinical impact. And PlumX wants to help researchers and institutions understand what is “impactful” in the clinical realm. So, we created a special category for references in clinical policy because of the history of clinical and translational science receiving less citations — and with that - less prestige and funding.
- #20 Clinical citations come from a variety of sources including PubMed Guidelines, Dynamed Plus and NICE (National Institute for Health Care) Clinical Guidelines in the UK. We will be adding Elsevier’s clinical key soon as well as looking for additional clinical sources to add.
- #21 Our current sources include PubMed guidelines, NICE guidelines from the UK and Dynamed Plus Topics (an EBSCO product). We’re adding ClinicalKey which is an Elsevier product as well.
- #22 PubMed practice guidelines are available as an article type.
- #23 Here’s another example of an article that has 4 clinical citations – 3 from PubMed and one from NICE in the UK. We link back to the guideline so you can see where the research is referenced.
- #24 If you are not familiar with DynaMed Plus, here a brief description. The content is developed using a strict, evidence-based editorial process, written by a team of specialized physicians and researchers with content being updated several times a day to include the latest evidence-based research.
- #25 When the Zika virus outbreak occurred in 2015 and hit North America, doctors were unprepared to diagnose and treat it. In 2015 there were 62 reported cases, but in 2016, there were 5,168 cases according to the Center for Disease Control which is part of the US National Institutes of Health. That’s a huge increase in a very short period of time. So doctors in hospitals and private practices needed to turn to sources like PubMed Guidelines or DynaMed Plus topics to be able to quickly diagnose and treat this rapidly spreading virus. This is an example where this article is cited in a DynaMed Plus topic.
- #26 Here’s what we link to from the PlumX Dashboard.
- #28 Another measure of societal impact is the Patent citation. Patent citations can demonstrate economic impact, as well as helping provide an understanding of how much research is being used in the creation of products.
- #29 Patent citations can illustrate a connection between the academic and corporate worlds and according to one article on the topic, can also show collaboration between not only researchers, but countries.
- #30 The Human Engineering Research Laboratories at the University of Pittsburgh’s mission is to “continuously improve the mobility and function of people with disabilities through advanced engineering in clinical research and medical rehabilitation” They want to create a world where all people with disabilities have unencumbered mobility and function so they can fully participate and contribute to society. While they publish a good amount of research in books, journals, conference proceedings, etc. they also research and develop all sorts of things that from wheelchairs that run on compressed air (versus a battery) to splints to treat hand injuries. In 2012 they received a patent for a joystick that allows individuals with substantial upper extremity motor impairments to have A wheelchair joystick with customized fit and function using mechanical and software customizations.
- #31 That joystick was patented. In PlumX, you can see that the joystick patent is being referenced by 10 other patents. All of this research contributing to the quality of life for those who are wheelchair-bound.
- #32 Clicking through to the US Patent and Trade Office database, you can see the other patents that reference it.
- #34 We look forward to your feedback and questions