Extending Sense-Making Models with Ideas from Cognition and Learning Theories
14 likes1,581 views
- The document proposes extending existing sense-making models with ideas from cognitive and learning theories to better understand the sense-making process. - A study was conducted with 6 participants to analyze their sense-making behaviors when completing intelligence analysis tasks using an information retrieval system. - The findings showed participants took an iterative, looping approach to sense-making involving exploration, focus, and knowledge updating. They primarily used data-driven cognitive mechanisms like extraction and comparison. Instantiated structures played an important role in sense-making.
![Findings – Dealing with Conflicts Disregard – “… I wanted that article to say something else. I have to disregard it.” Compromise – “…it is not my understanding that this has anything to do with oil. But okay, this is a different take.” Acceptance – “…I thought these countries had similar serious problems. But [in fact, they] seem to be relatively stable and put a lot of effort into establishing their economy.” Confusion – “…Obviously I have no idea what this is about...”](https://image.slidesharecdn.com/zhangasist08extendingsensemakingmodelsv3whiteonblack-1222273288070589-9/85/Extending-Sense-Making-Models-with-Ideas-from-Cognition-and-Learning-Theories-19-320.jpg)
![Pengyi Zhang [email_address] terpconnect.umd.edu/~pengyi/sensemaking/](https://image.slidesharecdn.com/zhangasist08extendingsensemakingmodelsv3whiteonblack-1222273288070589-9/85/Extending-Sense-Making-Models-with-Ideas-from-Cognition-and-Learning-Theories-23-320.jpg)
Extending Sense-Making Models with Ideas from Cognition and Learning Theories
- 1. Extending Sense-Making Models with Ideas from Cognition and Learning Theories Pengyi Zhang Dagobert Soergel Judith L. Klavans Douglas W. Oard
- 2. Outline What is sense-making? Theoretical framework A comprehensive sense-making model Pilot testing Conclusions Future work
- 3. Sense-making Sense-making is the process of creating an understanding of a problem or task so that further actions may be taken in an informed manner (Stefik et al., 1999).
- 4. Sense-making Sense-making is a pre-requisite for many other tasks such as decision making and problem solving; An important part of sense-making involves making clear the interrelated concepts and their relationships in a problem or task space.
- 5. A Sample Sense-making Scenario Task description An intelligence analyst is tasked to gather, analyze, and synthesize information related to Al-Bashir and to make recommendations for action in the form of a report. Background The Darfur conflict is a crisis in the Darfur region of western Sudan. Al-Bashir, the Sudanese president, is one of the key players in the area who is believed to have significant responsibility for continuous conflicts in the region. As part of an effort to resolve these armed conflicts, the administration needs to know as much as possible about al-Bashir in order to better negotiate with the involved parties and strategize its efforts.
- 7. Theoretical Framework Sense-making models Generic sense-making models Sense-making models of intelligence analysis Conducting research as sense-making Organizational sense-making Learning Schema theory Assimilation theory Generative Learning Theory Structural knowledge acquisition Cognition Types of conceptual changes Cognitive mechanisms Cognitive structures of knowledge Design of related tools Information extraction Representation and visualization tools Note-taking, analysis tools Task-specific and task-independent tools
- 8. Sense-making Elements Processes Sensing Making sense Acti vities Mechanisms Accretion Tuning Restructuring Inductive, data-driven Key item extraction Comparison Schema induction Generalization Deductive, structure-driven Definition Specification Elimination Explanation Inference Outcomes Identification of gaps Search Exploratory search Focused search for data for structure Building structure Instantiating structure Consuming instantiated structure Other Metaphor Classification Semantic fit Socratic dialogues
- 9. An Iterative Sense-making Model Data gap Structure gap Search: exploratory / focused Outcomes Updated knowledge Structure loop Data loop Identification of Gaps Searching for data Instantiating structure Accretion: Instantiated structure Tuning: Adapted structure Re-structuring: New structure Searching for structure Building structure Existing Knowledge Structures and their instantiations with data The iterations proceed from exploratory to focused search and sense-making. Task / Problem Decision / Solution / Task completion
- 10. Pilot Test Participants Data collection Data analysis Findings
- 11. Participants 6 information science students as surrogate subjects for intelligence analysts Tasked with synthesizing data from a variety of sources, assessing the credibility of information, and evaluating claims based on supporting evidence Trained in using Rosetta - a multilingual, multimedia news retrieval system
- 12. Data Collection 6 2-hour task sessions between March 2007 and May 2007, with 2 to 5 participants attending each Data consists of 17 think-aloud protocols – participants were instructed to verbalize their thoughts as they work on the tasks Sessions were logged and participants were interviewed after each session.
- 13. A Sample Task Scenario Task T1: al-Bashir (Abridged Version) Omar Hasan Ahmad al-Bashir is a Sudanese military leader, dictator, and current president of Sudan. Your task is to produce a report identifying information to assess the influence of al-Bashir as a basis for policy decisions and diplomatic actions. Requested information includes: key figures, organizations, and countries who have been associated with al-Bashir; his rise to power; and groups who have resisted him and the level of success in their resistance.
- 14. Data Analysis – Coding Scheme A Processes Search Sense-making Exploratory search Gap identification Exploratory search for data Data gap Exploratory search for structure Structural gap Focused search Building structures Focused search for data Using automatically extracted results Focused search for structure Extracting relationships manually Instantiating structures Updating knowledge B Conceptual Changes Sense-making success Sense-making failure Accretion Unable to fit data into structure Tuning Re-structuring Unable to build new structure
- 15. Data Analysis – Coding Scheme C Cognitive Mechanisms Inductive mechanisms Deductive mechanisms Key item extraction Definition Comparison Specification Similarity Explanation-based mechanisms Differentiation or discrimination Elimination Analogy and metaphor Semantic fit Classification Socratic dialogues Schema induction Inference Generalization D Emerging Codes Added During Analysis Reasons starting a new loop Resolution of conflicts New lead Disregard conflicting evidence Sense-making success Compromise Sense-making failure Accept new evidence Search failure Confusion
- 16. Sample Think-aloud Protocol with Coding Protocol (Energy Security Task) Loops Processes Conceptual Changes Cognitive Mechanisms Okay that was actually a very useful search. So let’s still take this query and look at Algeria, ‘cause obviously Algeria and Nigeria are very close… Loop 4 Focused Search for data Comparison I understand some of the keywords in the article but I don’t understand what the article… Sense-making Failure Key item extraction Okay this has to do with Algeria, southern Algeria. The minister of energy… OPEC meeting… so I am going to see what their connections are with OPEC. Building structures Key item extraction … with all the violence in Nigeria, I was expecting to find the same types of political outrage in Algeria… Instantiating structures Comparison and analogy and I’m not seeing any notice of that at all. Updating of knowledge Re-structuring
- 17. Findings – Search and Sense-making Loops The overall search and sense-making loops followed four stages: Task analysis exploratory stage focused stage updates of knowledge representation. Reasons for starting a new loop of search and sense-making: Success of previous sense-making Failure of previous sense-making New lead Failure of search
- 18. Findings – Cognitive Mechanisms Participants used a two-way approach: data-driven (bottom-up) 80% structure-driven (top-down) 20% Key item extraction and comparison were used most often.
- 19. Findings – Dealing with Conflicts Disregard – “… I wanted that article to say something else. I have to disregard it.” Compromise – “…it is not my understanding that this has anything to do with oil. But okay, this is a different take.” Acceptance – “…I thought these countries had similar serious problems. But [in fact, they] seem to be relatively stable and put a lot of effort into establishing their economy.” Confusion – “…Obviously I have no idea what this is about...”
- 20. Findings – Role of Instantiated Structures Entities (represented as names) and key concepts, (represented as keywords) were often the basis for relevance judgments. The relationships embedded in new information and between the new information and participants’ previous knowledge seemed to play an important role in structure building and data fitting. Both concepts and relationships seemed to be crucial for updating knowledge.
- 21. Conclusions The empirical think-aloud data seem to be consistent with the iterative sense-making model Users used mostly data-driven cognitive mechanisms Instantiated structure elements play an important role in sense-making
- 22. Implications and Future Work Theoretical Better understanding of sense-making processes by extending the existing models with ideas from cognition and learning Further testing and refinement. Empirical Better system design based on findings from user studies. Making design suggestions to support sense-making, not just search.
- 23. Pengyi Zhang [email_address] terpconnect.umd.edu/~pengyi/sensemaking/