Abstract image of a woman sitting on books and studying on a laptop

Graph's not dead: from unsupervised induction of linguistic structures from text towards applications in deep learning

Alexander Panchenko, profile picture
Alexander Panchenko

The document discusses the unsupervised induction of linguistic structures, focusing on word sense representations and their applications in deep learning. It outlines various methods for inducing word sense embeddings, semantic classes, and synsets, and mentions collaborations with multiple researchers in the field. Additionally, it highlights ongoing works and tasks related to word sense induction, including techniques like graph clustering and contextual analysis.

Science
Related topics:
Deep Learning • Natural Language Processing
1 of 124
Downloaded 14 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
Alexander Panchenko From unsupervised induction of linguistic structures from text towards applications in deep learning
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 2/98 In close collaboration with …
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 3/98 Andrei Kutuzov Eugen Ruppert Fide Marten Nikolay Arefyev Steffen Remus Martin Riedl Hubert Naets Maria Pelevina Anastasiya Lopukhina Konstantin Lopukhin In collaboration with …
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 4/98 Overview
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 5/98 Inducing word sense representations: word sense embeddings via retrofitting [Pelevina et al., 2016, Remus & Biemann, 2018]; inducing synsets [Ustalov et al., 2017b, Ustalov et al., 2017a, Ustalov et al., 2018b] inducing semantic classes [Panchenko et al., 2018b] Overview Overview
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 5/98 Inducing word sense representations: word sense embeddings via retrofitting [Pelevina et al., 2016, Remus & Biemann, 2018]; inducing synsets [Ustalov et al., 2017b, Ustalov et al., 2017a, Ustalov et al., 2018b] inducing semantic classes [Panchenko et al., 2018b] Making induced senses interpretable [Panchenko et al., 2017b, Panchenko et al., 2017c] Overview Overview
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 5/98 Inducing word sense representations: word sense embeddings via retrofitting [Pelevina et al., 2016, Remus & Biemann, 2018]; inducing synsets [Ustalov et al., 2017b, Ustalov et al., 2017a, Ustalov et al., 2018b] inducing semantic classes [Panchenko et al., 2018b] Making induced senses interpretable [Panchenko et al., 2017b, Panchenko et al., 2017c] Linking induced word senses to lexical resources [Panchenko, 2016, Faralli et al., 2016, Panchenko et al., 2017a, Biemann et al., 2018] Overview Overview
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 6/98 A shared task on word sense induction [Panchenko et al., 2018a, Arefyev et al., 2018] Overview Overview
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 6/98 A shared task on word sense induction [Panchenko et al., 2018a, Arefyev et al., 2018] Inducing semantic frames [Ustalov et al., 2018a] Inducing FrameNet-like structures; …using multi-way clustering. Overview Overview
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 6/98 A shared task on word sense induction [Panchenko et al., 2018a, Arefyev et al., 2018] Inducing semantic frames [Ustalov et al., 2018a] Inducing FrameNet-like structures; …using multi-way clustering. Learning graph/network embeddings [ongoing joint work with Andrei Kutuzov and Chris Biemann] How to represent induced networks/graphs? … so that they can be used in deep learning architectures. …effectively and efficiently. Overview Overview
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 7/98 Inducing word sense representations
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 8/98 Inducing word sense representations Word vs sense embeddings
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 9/98 Inducing word sense representations Word vs sense embeddings
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 10/98 Inducing word sense representations Related work
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 11/98 AutoExtend [Rothe & Schütze, 2015] * image is reproduced from the original paper Inducing word sense representations Related work: knowledge-based
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 12/98 Adagram [Bartunov et al., 2016] Multiple vector representations θ for each word: Inducing word sense representations Related work: knowledge-free
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 12/98 Adagram [Bartunov et al., 2016] Multiple vector representations θ for each word: p(Y, Z, β|X, α, θ) = V∏ w=1 ∞∏ k=1 p(βwk|α) N∏ i=1 [p(zi|xi, β) C∏ j=1 p(yij|zi, xi, θ zi – a hidden variable: a sense index of word xi in context C; α – a meta-parameter controlling number of senses. Inducing word sense representations Related work: knowledge-free
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 12/98 Adagram [Bartunov et al., 2016] Multiple vector representations θ for each word: p(Y, Z, β|X, α, θ) = V∏ w=1 ∞∏ k=1 p(βwk|α) N∏ i=1 [p(zi|xi, β) C∏ j=1 p(yij|zi, xi, θ zi – a hidden variable: a sense index of word xi in context C; α – a meta-parameter controlling number of senses. See also: [Neelakantan et al., 2014] and [Li and Jurafsky, 2015] Inducing word sense representations Related work: knowledge-free
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 13/98 Word sense induction (WSI) based on graph clustering: [Lin, 1998] [Pantel and Lin, 2002] [Widdows and Dorow, 2002] Chinese Whispers [Biemann, 2006] [Hope and Keller, 2013] Inducing word sense representations Related work: word sense induction
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 14/98 * source of the image: http://ic.pics.livejournal.com/blagin_anton/33716210/2701748/2701748_800.jpg Inducing word sense representations Related work: Chinese Whispers#1
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 15/98 Inducing word sense representations Related work: Chinese Whispers#2
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 16/98 Inducing word sense representations Related work: Chinese Whispers#2
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 17/98 Inducing word sense representations Related work: Chinese Whispers#2
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 18/98 RepL4NLP@ACL’16 [Pelevina et al., 2016], LREC’18 [Remus & Biemann, 2018] Prior methods: Induce inventory by clustering of word instances Use existing sense inventories Our method: Input: word embeddings Output: word sense embeddings Word sense induction by clustering of word ego-networks Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 19/98 From word embeddings to sense embeddings Calculate Word Similarity Graph Learning Word Vectors Word Sense Induction Text Corpus Word Vectors Word Similarity Graph Pooling of Word Vectors Sense Inventory Sense Vectors 1 2 4 3 Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 20/98 Word sense induction using ego-network clustering Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 21/98 Neighbours of Word and Sense Vectors Vector Nearest Neighbors table tray, bottom, diagram, bucket, brackets, stack, basket, list, parenthesis, cup, saucer, pile, playfield, bracket, pot, drop-down, cue, plate Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 21/98 Neighbours of Word and Sense Vectors Vector Nearest Neighbors table tray, bottom, diagram, bucket, brackets, stack, basket, list, parenthesis, cup, saucer, pile, playfield, bracket, pot, drop-down, cue, plate table#0 leftmost#0, column#1, tableau#1, indent#1, bracket#3, pointer#0, footer#1, cursor#1, diagram#0, grid#0 table#1 pile#1, stool#1, tray#0, basket#0, bowl#1, bucket#0, box#0, cage#0, saucer#3, mirror#1, pan#1, lid#0 Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 22/98 Word and sense embeddings of words iron and vitamin. LREC’18 [Remus & Biemann, 2018] Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 23/98 Word Sense Disambiguation 1 Context extraction: use context words around the target word 2 Context filtering: based on context word’s relevance for disambiguation 3 Sense choice in context: maximise similarity between a context vector and a sense vector Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 24/98 Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 25/98 Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 26/98 Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 27/98 Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 28/98 Unsupervised WSD SemEval’13, ReprL4NLP [Pelevina et al., 2016]: Model Jacc. Tau WNDCG F.NMI F.B-Cubed AI-KU (add1000) 0.176 0.609 0.205 0.033 0.317 AI-KU 0.176 0.619 0.393 0.066 0.382 AI-KU (remove5-add1000) 0.228 0.654 0.330 0.040 0.463 Unimelb (5p) 0.198 0.623 0.374 0.056 0.475 Unimelb (50k) 0.198 0.633 0.384 0.060 0.494 UoS (#WN senses) 0.171 0.600 0.298 0.046 0.186 UoS (top-3) 0.220 0.637 0.370 0.044 0.451 La Sapienza (1) 0.131 0.544 0.332 – – La Sapienza (2) 0.131 0.535 0.394 – – AdaGram, α = 0.05, 100 dim 0.274 0.644 0.318 0.058 0.470 w2v 0.197 0.615 0.291 0.011 0.615 w2v (nouns) 0.179 0.626 0.304 0.011 0.623 JBT 0.205 0.624 0.291 0.017 0.598 JBT (nouns) 0.198 0.643 0.310 0.031 0.595 TWSI (nouns) 0.215 0.651 0.318 0.030 0.573 Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 29/98 Semantic relatedness, LREC’2018 [Remus & Biemann, 2018]: autoextend adagram SGNS . glove . sympat . LSAbow . LSAhal . paragramSL . SimLex999 0.45 0.29 0.44 0.37 0.54 0.30 0.27 0.68 MEN 0.72 0.67 0.77 0.73 0.53 0.67 0.71 0.77 SimVerb 0.43 0.27 0.36 0.23 0.37 0.15 0.19 0.53 WordSim353 0.58 0.61 0.70 0.61 0.47 0.67 0.59 0.72 SimLex999-N 0.44 0.33 0.45 0.39 0.48 0.32 0.34 0.68 MEN-N 0.72 0.68 0.77 ____ 0.76 ____ 0.57 ____ 0.71 ____ 0.73 ____ 0.78 ____ Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 30/98 Unsupervised WSD SemEval’13, ReprL4NLP [Pelevina et al., 2016]: comparable to SOTA, incl. sense embeddings. Semantic relatedness, LREC’2018 [Remus & Biemann, 2018]: autoextend adagram SGNS SGNS+senses glove glove+senses sympat sympat+senses LSAbow LSAbow+senses LSAhal LSAhal+senses paragramSL paragramSL+senses SimLex999 0.45 0.29 0.44 0.46 0.37 0.41 0.54 0.55 0.30 0.39 0.27 0.38 0.68 0.64 MEN 0.72 0.67 0.77 0.78 0.73 0.77 0.53 0.68 0.67 0.70 0.71 0.74 0.77 0.80 SimVerb 0.43 0.27 0.36 0.39 0.23 0.30 0.37 0.45 0.15 0.22 0.19 0.28 0.53 0.53 WordSim353 0.58 0.61 0.70 0.69 0.61 0.65 0.47 0.62 0.67 0.66 0.59 0.63 0.72 0.73 SimLex999-N 0.44 0.33 0.45 0.50 0.39 0.47 0.48 0.55 0.32 0.46 0.34 0.44 0.68 0.66 MEN-N 0.72 0.68 0.77 0.79 0.76 0.80 0.57 0.74 0.71 0.73 0.73 0.76 0.78 0.81 Inducing word sense representations Sense embeddings using retrofitting
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 31/98 ACL’17 [Ustalov et al., 2017b] Examples of extracted synsets: Size Synset 2 {decimal point, dot} 3 {gullet, throat, food pipe} 4 {microwave meal, ready meal, TV dinner, frozen dinner} 5 {objective case, accusative case, oblique case, object case, accusative} 6 {radiotheater, dramatizedaudiobook, audiotheater, ra- dio play, radio drama, audio play} Inducing word sense representations Synset induction
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 32/98 Outline of the ’Watset’ method: Background Corpus Synonymy Dictionary Learning Word Embeddings Graph Construction Synsets Word Similarities Ambiguous Weighted Graph Local Clustering: Word Sense Induction Global Clustering: Synset Induction Sense Inventory Disambiguation of Neighbors Disambiguated Weighted Graph Local-Global Fuzzy Graph Clustering Inducing word sense representations Synset induction
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 33/98 Inducing word sense representations Synset induction
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 34/98 Inducing word sense representations Synset induction
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 35/98 Inducing word sense representations Synset induction
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 36/98 CW MCL MaxMax ECO CPM Watset 0.0 0.1 0.2 0.3 WordNet (English) F−score CW MCL MaxMax ECO CPM Watset 0.0 0.1 0.2 0.3 BabelNet (English) F−score Inducing word sense representations Synset induction
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 36/98 CW MCL MaxMax ECO CPM Watset 0.0 0.1 0.2 0.3 WordNet (English) F−score CW MCL MaxMax ECO CPM Watset 0.0 0.1 0.2 0.3 BabelNet (English) F−score CW MCL MaxMax ECO CPM Watset 0.00 0.05 0.10 0.15 0.20 RuWordNet (Russian) F−score CW MCL MaxMax ECO CPM Watset 0.0 0.1 0.2 0.3 0.4 YARN (Russian) F−score Inducing word sense representations Synset induction
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 37/98 Word Sense Local Sense Cluster: Related Senses Hypernyms mango#0 peach#1, grape#0, plum#0, apple#0, apricot#0, watermelon#1, banana#1, coconut#0, pear#0, fig#0, melon#0, mangosteen#0, … fruit#0, food#0, … apple#0 mango#0, pineapple#0, banana#1, melon#0, grape#0, peach#1, watermelon#1, apricot#0, cranberry#0, pumpkin#0, mangosteen#0, … fruit#0, crop#0, … Java#1 C#4, Python#3, Apache#3, Ruby#6, Flash#1, C++#0, SQL#0, ASP#2, Visual Basic#1, CSS#0, Delphi#2, MySQL#0, Excel#0, Pascal#0, … programming language#3, lan- guage#0, … Python#3 PHP#0, Pascal#0, Java#1, SQL#0, Visual Ba- sic#1, C++#0, JavaScript#0, Apache#3, Haskell#5, .NET#1, C#4, SQL Server#0, … language#0, tech- nology#0, … Inducing word sense representations Sample of induced sense inventory
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 38/98 ID Global Sense Cluster: Semantic Class Hypernyms 1 peach#1, banana#1, pineapple#0, berry#0, black- berry#0, grapefruit#0, strawberry#0, blueberry#0, mango#0, grape#0, melon#0, orange#0, pear#0, plum#0, raspberry#0, watermelon#0, apple#0, apri- cot#0, watermelon#0, pumpkin#0, berry#0, man- gosteen#0, … vegetable#0, fruit#0, crop#0, ingredi- ent#0, food#0, · 2 C#4, Basic#2, Haskell#5, Flash#1, Java#1, Pas- cal#0, Ruby#6, PHP#0, Ada#1, Oracle#3, Python#3, Apache#3, Visual Basic#1, ASP#2, Delphi#2, SQL Server#0, CSS#0, AJAX#0, JavaScript#0, SQL Server#0, Apache#3, Delphi#2, Haskell#5, .NET#1, CSS#0, … programming lan- guage#3, technol- ogy#0, language#0, format#2, app#0 Inducing word sense representations Sample of induced semantic classes
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 39/98 Text Corpus Representing Senses with Ego Networks Semantic Classes Word Sense Induction from Text Corpus Sense Graph Construction Clustering of Word Senes Labeling Sense Clusters with Hypernyms Induced Word Senses Sense Ego-Networks Global Sense Graph s Noisy Hypernyms Cleansed Hypernyms Induction of Semantic Classes Global Sense Clusters Inducing word sense representations Induction of semantic classes
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 40/98 Filtering noisy hypernyms with semantic classes LREC’18 [Panchenko et al., 2018b]: fruit#1 food#0 apple#2 mango#0 pear#0 Hypernyms, Sense Cluster, mangosteen#0 city#2 Removed Wrong Added Missing Inducing word sense representations Induction of sense semantic classes
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 41/98 http://panchenko.me/data/joint/nodes20000-layers7 Inducing word sense representations Global sense clustering
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 42/98 Inducing word sense representations Global sense clustering
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 43/98 Filtering of a noisy hypernymy database with semantic classes. LREC’18 [Panchenko et al., 2018b] Precision Recall F-score Original Hypernyms (Seitner et al., 2016) 0.475 0.546 0.508 Semantic Classes (coarse-grained) 0.541 0.679 0.602 Inducing word sense representations Induction of sense semantic classes
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 44/98 Making induced senses interpretable
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 45/98 Knowledge-based sense representations are interpretable Making induced senses interpretable Making induced senses interpretable
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 46/98 Most knowledge-free sense representations are uninterpretable Making induced senses interpretable Making induced senses interpretable
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 47/98 Making induced senses interpretable Making induced senses interpretable
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 48/98 Hypernymy prediction in context. EMNLP’17 [Panchenko et al., 2017b] Making induced senses interpretable Making induced senses interpretable
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 49/98 11.702 sentences, 863 words with avg.polysemy of 3.1. WSD Model Accuracy Inventory Features Hypers HyperHypers Word Senses Random 0.257 0.610 Word Senses MFS 0.292 0.682 Word Senses Cluster Words 0.291 0.650 Word Senses Context Words 0.308 0.686 Making induced senses interpretable Making induced senses interpretable
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 49/98 11.702 sentences, 863 words with avg.polysemy of 3.1. WSD Model Accuracy Inventory Features Hypers HyperHypers Word Senses Random 0.257 0.610 Word Senses MFS 0.292 0.682 Word Senses Cluster Words 0.291 0.650 Word Senses Context Words 0.308 0.686 Super Senses Random 0.001 0.001 Super Senses MFS 0.001 0.001 Super Senses Cluster Words 0.174 0.365 Super Senses Context Words 0.086 0.188 Making induced senses interpretable Making induced senses interpretable
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 50/98 Linking induced senses to resources
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 51/98 Text Corpus Linking Induced Senses to Senses of the LR Induce a Graph of Sem. Related Words Enriched Lexical Resource Graph of Related Words Word Sense Induction Labeling Senses with Hypernyms Disambiguation of Neighbours Typing of the Unmapped Induced Senses Word Sense Inventory Labeled Word Senses PCZ Construction of Proto-Conceptualization (PCZ) Linking Proto-Conceptualization to Lexical Resource Part. Linked Senses to the LR Lexical Resource (LR): WordNet, BabelNet, ... Construction of sense feature representations Graph of Related Senses LREC’16 [Panchenko, 2016], ISWC’16 [Faralli et al., 2016], SENSE@EACL’17 [Panchenko et al., 2017a], NLE’18 [Biemann et al., 2018] Linking induced senses to resources Linking induced senses to resources
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 52/98 Word AdaGram BabelNet AdaGram BoW BabelNet BoW python 2 bn:01713224n perl, php, java, smalltalk, ruby, lua, tcl, scripting, javascript, bindings, binding, programming, coldfusion, actionscript, net, . . . language, programming, python- ista, python programming, python3, python2, level, com- puter, pythonistas, python3000, python 1 bn:01157670n monty, circus, spamalot, python, magoo, muppet, snoopy, fea- turette, disney, tunes, tune, clas- sic, shorts, short, apocalypse, . . . monty, comedy, monty python, british, monte, monte python, troupe, pythonesque, foot, artist, record, surreal, terry, . . . python 3 bn:00046456n spectacled, unicornis, snake, gi- ant, caiman, leopard, squirrel, crocodile, horned, cat, mole, ele- phant, opossum, pheasant, . . . molurus, indian, boa, tigris, tiger python, rock, tiger, indian python, reptile, python molurus, indian rock python, coluber, . . . python 4 bn:01157670n circus, fly, flying, dusk, lizard, moth, unicorn, puff, adder, vul- ture, tyrannosaurus, zephyr, bad- ger, . . . monty, comedy, monty python, british, monte, monte python, troupe, pythonesque, foot, artist, record, surreal, terry, . . . python 1 bn:00473212n monty, circus, spamalot, python, magoo, muppet, snoopy, fea- turette, disney, tunes, tune, clas- sic, shorts, short, apocalypse, . . . pictures, monty, python monty pictures, limited, company, python pictures limited, king- dom, picture, serve, director, . . . python 1 bn:03489893n monty, circus, spamalot, python, magoo, muppet, snoopy, fea- turette, disney, tunes, tune, clas- sic, shorts, short, apocalypse, . . . film, horror, movie, clabaugh, richard, monster, century, direct, snake, python movie, television, giant, natural, language, for-tv, . . . Linking induced senses to resources Linking induced senses to resources
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 53/98 Model Representation of the Sense ”disk (medium)” WordNet memory, device, floppy, disk, hard, disk, disk, computer, science, computing, diskette, fixed, disk, floppy, magnetic, disc, magnetic, disk, hard, disc, storage, device WordNet + Linked recorder, disk, floppy, console, diskette, handset, desktop, iPhone, iPod, HDTV, kit, RAM, Discs, Blu- ray, computer, GB, microchip, site, cartridge, printer, tv, VCR, Disc, player, LCD, software, component, camcorder, cellphone, card, monitor, display, burner, Web, stereo, internet, model, iTunes, turntable, chip, cable, camera, iphone, notebook, device, server, surface, wafer, page, drive, laptop, screen, pc, television, hardware, YouTube, dvr, DVD, product, folder, VCR, radio, phone, circuitry, partition, megabyte, peripheral, format, machine, tuner, website, merchandise, equipment, gb, discs, MP3, hard-drive, piece, video, storage device, memory device, microphone, hd, EP, content, soundtrack, webcam, system, blade, graphic, microprocessor, collection, document, programming, battery, key- board, HD, handheld, CDs, reel, web, material, hard-disk, ep, chart, debut, configuration, recording, album, broadcast, download, fixed disk, planet, pda, microfilm, iPod, videotape, text, cylinder, cpu, canvas, label, sampler, workstation, electrode, magnetic disc, catheter, magnetic disk, Video, mo- bile, cd, song, modem, mouse, tube, set, ipad, signal, substrate, vinyl, music, clip, pad, audio, com- pilation, memory, message, reissue, ram, CD, subsystem, hdd, touchscreen, electronics, demo, shell, sensor, file, shelf, processor, cassette, extra, mainframe, motherboard, floppy disk, lp, tape, version, kilobyte, pacemaker, browser, Playstation, pager, module, cache, DVD, movie, Windows, cd-rom, e- book, valve, directory, harddrive, smartphone, audiotape, technology, hard disk, show, computing, computer science, Blu-Ray, blu-ray, HDD, HD-DVD, scanner, hard disc, gadget, booklet, copier, play- back, TiVo, controller, filter, DVDs, gigabyte, paper, mp3, CPU, dvd-r, pipe, cd-r, playlist, slot, VHS, film, videocassette, interface, adapter, database, manual, book, channel, changer, storage Linking induced senses to resources Linking induced senses to resources
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 54/98 Evaluation of linking accuracy: Linking induced senses to resources Linking induced senses to resources
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 55/98 Evaluation of enriched representations based on WSD: Linking induced senses to resources Linking induced senses to resources
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 56/98 Shared task on word sense induction
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 57/98 An ACL SIGSLAV sponsored shared task on word sense induction (WSI) for the Russian language. More details: https://russe.nlpub.org/2018/wsi Shared task on word sense induction A shared task on WSI
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 58/98 Target word, e.g. “bank”. Shared task on word sense induction A lexical sample WSI task
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 58/98 Target word, e.g. “bank”. Contexts where the word occurs, e.g.: “river bank is a slope beside a body of water” “bank is a financial institution that accepts deposits” “Oh, the bank was robbed. They took about a million dollars.” “bank of Elbe is a good and popular hangout spot complete with good food and fun” Shared task on word sense induction A lexical sample WSI task
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 58/98 Target word, e.g. “bank”. Contexts where the word occurs, e.g.: “river bank is a slope beside a body of water” “bank is a financial institution that accepts deposits” “Oh, the bank was robbed. They took about a million dollars.” “bank of Elbe is a good and popular hangout spot complete with good food and fun” You need to group the contexts by senses: “river bank is a slope beside a body of water” “bank of Elbe is a good and popular hangout spot complete with good food and fun” “bank is a financial institution that accepts deposits” “Oh, the bank was robbed. They took about a million dollars.” Shared task on word sense induction A lexical sample WSI task
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 59/98 Shared task on word sense induction Dataset based on Wikipedia
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 60/98 Shared task on word sense induction Dataset based on RNC
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 61/98 Shared task on word sense induction Dataset based on dictionary glosses
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 62/98 Shared task on word sense induction A sample from the wiki-wiki dataset
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 63/98 Shared task on word sense induction A sample from the wiki-wiki dataset
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 64/98 Shared task on word sense induction A sample from the wiki-wiki dataset
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 65/98 Shared task on word sense induction A sample from the bts-rnc dataset
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 66/98 Shared task on word sense induction A sample from the active-dict dataset
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 67/98 1 Get the neighbors of a target word, e.g. “bank”: 1 lender 2 river 3 citybank 4 slope 5 … 2 Get similar to “bank” and dissimilar to “lender”: 1 river 2 slope 3 land 4 … 3 Compute distances to “lender” and “river”. Shared task on word sense induction jamsic: sense induction
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 68/98 Induction of semantic frames
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 69/98 Induction of semantic frames FrameNet: frame “Kidnapping”
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 70/98 ACL’2018 [Ustalov et al., 2018a] Example of a LU tricluster corresponding to the “Kidnapping” frame from FrameNet. FrameNet Role Lexical Units (LU) Perpetrator Subject kidnapper, alien, militant FEE Verb snatch, kidnap, abduct Victim Object son, people, soldier, child Induction of semantic frames Frame induction as a triclustering
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 71/98 Induction of semantic frames SVO triple elements
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 72/98 Officer|chair|Committee officer|head|team mayor|lead|city officer|lead|company Mayor|lead|city boss|lead|company chairman|lead|company director|lead|department chief|lead|department president|lead|government president|lead|state director|lead|company president|lead|department officer|chair|committee Chief|lead|department chairman|lead|committee Director|lead|Department Director|lead|department Director|lead|agency Director|lead|company minister|lead|team Director|head|team director|head|team Chairman|lead|company Chairman|lead|Committee President|lead|company Director|chair|Committee President|lead|party President|head|teamleader|head|team Director|chair|committee director|chair|committee Director|head|Department president|head|team director|head|department director|head|agency director|head|committee Chairman|run|committee Chairman|chair|Committee Chairman|chair|committee President|chair|Committee President|chair|committee Governor|lead|state chairman|head|committee chairman|run|committee president|chair|committee president|head|committee president|chair|Committee Minister|chair|committee representative|chair|committee representative|head|committee General|command|department General|command|Department General|head|Department General|head|department officer|head|department minister|head|department leader|head|agency leader|head|party leader|head|committee leader|head|department minister|head|committee King|run|company leader|head|government Minister|head|government president|head|government Officer|chair|Committee officer|head|team mayor|lead|city officer|lead|company Mayor|lead|city boss|lead|company chairman|lead|company director|lead|department chief|lead|department president|lead|government president|lead|state director|lead|company president|lead|department officer|chair|committee Chief|lead|department chairman|lead|committee Director|lead|Department Director|lead|department Director|lead|agency Director|lead|company minister|lead|team Director|head|team director|head|team Chairman|lead|company Chairman|lead|Committee President|lead|company Director|chair|Committee President|lead|party President|head|teamleader|head|team Director|chair|committee director|chair|committee Director|head|Department president|head|team director|head|department director|head|agency director|head|committee Chairman|run|committee Chairman|chair|Committee Chairman|chair|committee President|chair|Committee President|chair|committee Governor|lead|state chairman|head|committee chairman|run|committee president|chair|committee president|head|committee president|chair|Committee Minister|chair|committee representative|chair|committee representative|head|committee General|command|department General|command|Department General|head|Department General|head|department officer|head|department minister|head|department leader|head|agency leader|head|party leader|head|committee leader|head|department minister|head|committee King|run|company leader|head|government Minister|head|government president|head|government Induction of semantic frames An SVO triple graph
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 73/98 Input: an embedding model v ∈ V → ⃗v ∈ Rd, a set of SVO triples T ⊆ V 3, the number of nearest neighbors k ∈ N, a graph clustering algorithm Cluster. Induction of semantic frames Triframes frame induction
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 73/98 Input: an embedding model v ∈ V → ⃗v ∈ Rd, a set of SVO triples T ⊆ V 3, the number of nearest neighbors k ∈ N, a graph clustering algorithm Cluster. Output: a set of triframes F. Induction of semantic frames Triframes frame induction
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 73/98 Input: an embedding model v ∈ V → ⃗v ∈ Rd, a set of SVO triples T ⊆ V 3, the number of nearest neighbors k ∈ N, a graph clustering algorithm Cluster. Output: a set of triframes F. S ← {t→ ⃗t ∈ R3d : t ∈ T} E ← {(t, t′) ∈ T2 : t′ ∈ NNS k (⃗t), t ̸= t′} F ← ∅ for all C ∈ Cluster(T, E) do fs ← {s ∈ V : (s, v, o) ∈ C} fv ← {v ∈ V : (s, v, o) ∈ C} fo ← {o ∈ V : (s, v, o) ∈ C} F ← F ∪ {(fs, fv, fo)} return F Induction of semantic frames Triframes frame induction
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 74/98 Frame # 848 Subjects: Company, firm, company Verbs: buy, supply, discharge, purchase, expect Objects: book, supply, house, land, share, company, grain, which, item, product, ticket, work, this, equipment, House, it, film, water, something, she, what, service, plant, time Induction of semantic frames Example of an extracted frame
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 75/98 Frame # 849 Subjects: student, scientist, we, pupil, member, company, man, nobody, you, they, US, group, it, people, Man, user, he Verbs: do, test, perform, execute, conduct Objects: experiment, test Induction of semantic frames Example of an extracted frame
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 76/98 Frame # 3207 Subjects: people, we, they, you Verbs: feel, seek, look, search Objects: housing, inspiration, gold, witness, part- ner, accommodation, Partner Induction of semantic frames Example of an extracted frame
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 77/98 Dataset # instances # unique # clusters FrameNet Triples 99,744 94,170 383 Poly. Verb Classes 246 110 62 Induction of semantic frames Evaluation datasets
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 78/98 Dataset # instances # unique # clusters FrameNet Triples 99,744 94,170 383 Poly. Verb Classes 246 110 62 Induction of semantic frames Evaluation settings
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 78/98 Dataset # instances # unique # clusters FrameNet Triples 99,744 94,170 383 Poly. Verb Classes 246 110 62 Quality Measures: nmPU: normalized modified purity, niPU: normalized inverse purity. Induction of semantic frames Evaluation settings
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 79/98 F1-scores for verbs, subjects, objects, frames Induction of semantic frames Results: comparison to state-of-art
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 80/98 Graph embeddings
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 81/98 Graph embeddings Text: sparse symbolic representation
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 81/98 Image source: https://www.tensorflow.org/tutorials/word2vec Graph embeddings Text: sparse symbolic representation
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 82/98 Graph embeddings Graph: sparse symbolic representation
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 83/98 From a survey on graph embeddings [Hamilton et al., 2017]: Graph embeddings Embedding graph into a vector space
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 84/98 From a survey on graph embeddings [Hamilton et al., 2017]: Graph embeddings Learning with an “autoencoder”
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 85/98 From a survey on graph embeddings [Hamilton et al., 2017]: Graph embeddings Some established approaches
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 86/98 An submitted joint work with Andrei Kutuzov and Chris Biemann: Given a tree (V, E) Leackock-Chodorow (LCH) similarity measure: sim(vi, vj) = − log shortest_path_distance(vi, vj) 2h Graph embeddings Graph embeddings using similarities
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 86/98 An submitted joint work with Andrei Kutuzov and Chris Biemann: Given a tree (V, E) Leackock-Chodorow (LCH) similarity measure: sim(vi, vj) = − log shortest_path_distance(vi, vj) 2h Jiang-Conrath (JCN) similarity measure: sim(vi, vj) = 2 ln Plcs(vi, vj) ln P(vi) + ln P(vj) Graph embeddings Graph embeddings using similarities
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 87/98 path2vec: Approximating Structural Node Similarities with Node Embeddings: J = 1 |T| ∑ (vi,vj)∈T (vi · vj − sim(vi, vj))2 , where: sim(vi, vj) - the value of a ‘gold’ similarity measure between a pair of nodes vi and vj; vi - an embeddings of node; T - training batch. Graph embeddings Graph embeddings using similarities
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 88/98 Computation of 82,115 pairwise similarities: Model Running time LCH in NLTK 30 sec. JCN in NLTK 6.7 sec. FSE embeddings 0.713 sec. path2vec and other float vectors 0.007 sec. Graph embeddings Speedup: graph vs embeddings
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 89/98 Spearman correlation scores with WordNet similarities on SimLex999 noun pairs: Selection of synsets Model JCN-SemCor JCN-Brown LCH WordNet 1.0 1.0 1.0 Node2vec 0.655 0.671 0.724 Deepwalk 0.775 0.774 0.868 FSE 0.830 0.820 0.900 path2vec 0.917 0.914 0.934 Graph embeddings Results: goodness of fit
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 90/98 Spearman correlations with human SimLex999 noun similarities: Model Correlation Raw WordNet JCN-SemCor 0.487 Raw WordNet JCN-Brown 0.495 Raw WordNet LCH 0.513 node2vec [Grover & Leskovec, 2016] 0.450 Deepwalk [Perozzi et al., 2014] 0.533 FSE [Subercaze et al., 2015] 0.556 path2vec JCN-SemCor 0.526 path2vec JCN-Brown 0.487 path2vec LCH 0.522 Graph embeddings Results: SimLex999 dataset
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 91/98 JCN (left) and LCH (right): Graph embeddings Results: SimLex999 dataset
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 92/98 WSD: each column lists all the possible synsets for the corresponding word. Graph embeddings Results: word sense disambiguation
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 93/98 F1 scores on Senseval-2 word sense disambiguation task: Model F-measure WordNet JCN-SemCor 0.620 WordNet JCN-Brown 0.561 WordNet LCH 0.547 node2vec [Grover & Leskovec, 2016] 0.501 Deepwalk [Perozzi et al., 2014] 0.528 FSE [Subercaze et al., 2015] 0.536 path2vec Batch size: 20 50 100 JCN-SemCor 0.543 0.543 0.535 JCN-Brown 0.538 0.515 0.542 LCH 0.540 0.535 0.536 Graph embeddings Results: word sense disambiguation
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 94/98 Conclusion
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 95/98 Conclusion Vectors + Graphs = ♡
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 96/98 We can induce word senses, synsets, semantic classes, and semantic frames in a knowledge-free way using graph clustering and distributional models. Conclusion Take home messages
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 96/98 We can induce word senses, synsets, semantic classes, and semantic frames in a knowledge-free way using graph clustering and distributional models. We can make the induced word senses interpretable in a knowledge-free way with hypernyms, images, definitions. Conclusion Take home messages
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 96/98 We can induce word senses, synsets, semantic classes, and semantic frames in a knowledge-free way using graph clustering and distributional models. We can make the induced word senses interpretable in a knowledge-free way with hypernyms, images, definitions. We can link induced senses to lexical resources to improve performance of WSD; enrich lexical resources with emerging senses. Conclusion Take home messages
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 96/98 We can induce word senses, synsets, semantic classes, and semantic frames in a knowledge-free way using graph clustering and distributional models. We can make the induced word senses interpretable in a knowledge-free way with hypernyms, images, definitions. We can link induced senses to lexical resources to improve performance of WSD; enrich lexical resources with emerging senses. We can represent language graphs using graph embeddings in deep neural models. Conclusion Take home messages
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 97/98 A special issue on informing neural architectures for NLP with linguistic and background knowledge. … with Ivan Vulić and Simone Paolo Ponzetto. goo.gl/A76NGX Conclusion Natural Language Engineering journal
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 Conclusion Thank you! Questions?
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 Arefyev, N., Ermolaev, P., & Panchenko, A. (2018). How much does a word weigh? weighting word embeddings for word sense induction. arXiv preprint arXiv:1805.09209. Bartunov, S., Kondrashkin, D., Osokin, A., & Vetrov, D. (2016). Breaking sticks and ambiguities with adaptive skip-gram. In Artificial Intelligence and Statistics (pp. 130–138). Biemann, C., Faralli, S., Panchenko, A., & Ponzetto, S. P. (2018). A framework for enriching lexical semantic resources with distributional semantics. In Journal of Natural Language Engineering (pp. 56–64).: Cambridge Press. Faralli, S., Panchenko, A., Biemann, C., & Ponzetto, S. P. (2016). Linked disambiguated distributional semantic networks. In International Semantic Web Conference (pp. 56–64).: Springer. Grover, A. & Leskovec, J. (2016).
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 Node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 855–864).: ACM. Hamilton, W. L., Ying, R., & Leskovec, J. (2017). Representation learning on graphs: Methods and applications. IEEE Data Engineering Bulletin, September 2017. Panchenko, A. (2016). Best of both worlds: Making word sense embeddings interpretable. In LREC. Panchenko, A., Faralli, S., Ponzetto, S. P., & Biemann, C. (2017a). Using linked disambiguated distributional networks for word sense disambiguation.
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 In Proceedings of the 1st Workshop on Sense, Concept and Entity Representations and their Applications (pp. 72–78). Valencia, Spain: Association for Computational Linguistics. Panchenko, A., Lopukhina, A., Ustalov, D., Lopukhin, K., Arefyev, N., Leontyev, A., & Loukachevitch, N. (2018a). Russe’2018: A shared task on word sense induction for the russian language. arXiv preprint arXiv:1803.05795. Panchenko, A., Marten, F., Ruppert, E., Faralli, S., Ustalov, D., Ponzetto, S. P., & Biemann, C. (2017b). Unsupervised, knowledge-free, and interpretable word sense disambiguation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (pp. 91–96). Copenhagen, Denmark: Association for Computational Linguistics. Panchenko, A., Ruppert, E., Faralli, S., Ponzetto, S. P., & Biemann, C. (2017c).
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 Unsupervised does not mean uninterpretable: The case for word sense induction and disambiguation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers (pp. 86–98). Valencia, Spain: Association for Computational Linguistics. Panchenko, A., Ustalov, D., Faralli, S., Ponzetto, S. P., & Biemann, C. (2018b). Improving hypernymy extraction with distributional semantic classes. In Proceedings of the LREC 2018 Miyazaki, Japan: European Language Resources Association. Pelevina, M., Arefiev, N., Biemann, C., & Panchenko, A. (2016). Making sense of word embeddings. In Proceedings of the 1st Workshop on Representation Learning for NLP (pp. 174–183). Berlin, Germany: Association for Computational Linguistics. Perozzi, B., Al-Rfou, R., & Skiena, S. (2014).
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 701–710).: ACM. Remus, S. & Biemann, C. (2018). Retrofittingword representations for unsupervised sense aware word similarities. In Proceedings of the LREC 2018 Miyazaki, Japan: European Language Resources Association. Rothe, S. & Schütze, H. (2015). Autoextend: Extending word embeddings to embeddings for synsets and lexemes. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 1793–1803). Beijing, China: Association for Computational Linguistics. Subercaze, J., Gravier, C., & Laforest, F. (2015).
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 On metric embedding for boosting semantic similarity computations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (pp. 8–14).: Association for Computational Linguistics. Ustalov, D., Chernoskutov, M., Biemann, C., & Panchenko, A. (2017a). Fighting with the sparsity of synonymy dictionaries for automatic synset induction. In International Conference on Analysis of Images, Social Networks and Texts (pp. 94–105).: Springer. Ustalov, D., Panchenko, A., & Biemann, C. (2017b). Watset: Automatic induction of synsets from a graph of synonyms. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp.
May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 1579–1590). Vancouver, Canada: Association for Computational Linguistics. Ustalov, D., Panchenko, A., Kutuzov, A., Biemann, C., & Ponzetto, S. P. (2018a). Unsupervised semantic frame induction using triclustering. arXiv preprint arXiv:1805.04715. Ustalov, D., Teslenko, D., Panchenko, A., Chernoskutov, M., & Biemann, C. (2018b). Word sense disambiguation based on automatically induced synsets. In LREC 2018, 11th International Conference on Language Resources and Evaluation : 7-12 May 2018, Miyazaki (Japan) (pp. tba). Paris: European Language Resources Association, ELRA-ELDA. Accepted for publication.

More Related Content

PDF
The importance of explicit teaching of verbs and interrogative utterances in ...
Nordiskt Nätverk för Vuxnas Lärande
 
PDF
Inducing Interpretable Word Senses for WSD and Enrichment of Lexical Resources
Alexander Panchenko
 
PDF
Making Sense of Word Embeddings
Alexander Panchenko
 
PPTX
Using Text Embeddings for Information Retrieval
Bhaskar Mitra
 
PPTX
From Natural Language Processing to Artificial Intelligence
Jonathan Mugan
 
PDF
AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transforma...
Tomoyuki Suzuki
 
PPTX
Data Day Seattle, From NLP to AI
Jonathan Mugan
 
PPTX
Deep Neural Methods for Retrieval
Bhaskar Mitra
 
The importance of explicit teaching of verbs and interrogative utterances in ...
Nordiskt Nätverk för Vuxnas Lärande
 
Inducing Interpretable Word Senses for WSD and Enrichment of Lexical Resources
Alexander Panchenko
 
Making Sense of Word Embeddings
Alexander Panchenko
 
Using Text Embeddings for Information Retrieval
Bhaskar Mitra
 
From Natural Language Processing to Artificial Intelligence
Jonathan Mugan
 
AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transforma...
Tomoyuki Suzuki
 
Data Day Seattle, From NLP to AI
Jonathan Mugan
 
Deep Neural Methods for Retrieval
Bhaskar Mitra
 

Similar to Graph's not dead: from unsupervised induction of linguistic structures from text towards applications in deep learning (20)

PDF
Noun Sense Induction and Disambiguation using Graph-Based Distributional Sema...
Alexander Panchenko
 
PDF
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
Matthew Lease
 
PDF
Deep learning based drug protein interaction
NAVER Engineering
 
PDF
StarSpace: Embed All The Things!
☕ Keita Watanabe
 
PDF
Text Representations for Deep learning
Zachary S. Brown
 
PDF
Should we be afraid of Transformers?
Dominik Seisser
 
PPTX
Text Mining for Lexicography
Leiden University
 
PPTX
Designing, Visualizing and Understanding Deep Neural Networks
connectbeubax
 
PPTX
A Simple Introduction to Word Embeddings
Bhaskar Mitra
 
PDF
(Deep) Neural Networks在 NLP 和 Text Mining 总结
君 廖
 
PDF
Deep Learning for Information Retrieval
Roelof Pieters
 
PPTX
Lecture1.pptx
jonathanG19
 
PPTX
Vectorland: Brief Notes from Using Text Embeddings for Search
Bhaskar Mitra
 
PPTX
Neural Models for Information Retrieval
Bhaskar Mitra
 
PPTX
The Neural Search Frontier - Doug Turnbull, OpenSource Connections
Lucidworks
 
PPTX
Vectors in Search - Towards More Semantic Matching
Simon Hughes
 
PPTX
Vectors in Search – Towards More Semantic Matching - Simon Hughes, Dice.com
Lucidworks
 
PPTX
Word vectors
Adwait Bhave
 
PPTX
A Simple Introduction to Neural Information Retrieval
Bhaskar Mitra
 
PDF
Yun-Nung (Vivian) Chen - 2015 - Matrix Factorization with Knowledge Graph Pr...
Association for Computational Linguistics
 
Noun Sense Induction and Disambiguation using Graph-Based Distributional Sema...
Alexander Panchenko
 
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
Matthew Lease
 
Deep learning based drug protein interaction
NAVER Engineering
 
StarSpace: Embed All The Things!
☕ Keita Watanabe
 
Text Representations for Deep learning
Zachary S. Brown
 
Should we be afraid of Transformers?
Dominik Seisser
 
Text Mining for Lexicography
Leiden University
 
Designing, Visualizing and Understanding Deep Neural Networks
connectbeubax
 
A Simple Introduction to Word Embeddings
Bhaskar Mitra
 
(Deep) Neural Networks在 NLP 和 Text Mining 总结
君 廖
 
Deep Learning for Information Retrieval
Roelof Pieters
 
Lecture1.pptx
jonathanG19
 
Vectorland: Brief Notes from Using Text Embeddings for Search
Bhaskar Mitra
 
Neural Models for Information Retrieval
Bhaskar Mitra
 
The Neural Search Frontier - Doug Turnbull, OpenSource Connections
Lucidworks
 
Vectors in Search - Towards More Semantic Matching
Simon Hughes
 
Vectors in Search – Towards More Semantic Matching - Simon Hughes, Dice.com
Lucidworks
 
Word vectors
Adwait Bhave
 
A Simple Introduction to Neural Information Retrieval
Bhaskar Mitra
 
Yun-Nung (Vivian) Chen - 2015 - Matrix Factorization with Knowledge Graph Pr...
Association for Computational Linguistics
 
Ad

More from Alexander Panchenko (18)

PDF
Building a Web-Scale Dependency-Parsed Corpus from Common Crawl
Alexander Panchenko
 
PDF
Improving Hypernymy Extraction with Distributional Semantic Classes
Alexander Panchenko
 
PDF
IIT-UHH at SemEval-2017 Task 3: Exploring Multiple Features for Community Que...
Alexander Panchenko
 
PDF
Fighting with Sparsity of the Synonymy Dictionaries for Automatic Synset Indu...
Alexander Panchenko
 
PDF
The 6th Conference on Analysis of Images, Social Networks, and Texts (AIST 2...
Alexander Panchenko
 
PDF
Using Linked Disambiguated Distributional Networks for Word Sense Disambiguation
Alexander Panchenko
 
PDF
Unsupervised Does Not Mean Uninterpretable: The Case for Word Sense Induction...
Alexander Panchenko
 
PPTX
Getting started in Apache Spark and Flink (with Scala) - Part II
Alexander Panchenko
 
PPTX
IIT-TUDA at SemEval-2016 Task 5: Beyond Sentiment Lexicon: Combining Domain ...
Alexander Panchenko
 
PDF
Text Analysis of Social Networks: Working with FB and VK Data
Alexander Panchenko
 
PPTX
Неологизмы в социальной сети Фейсбук
Alexander Panchenko
 
PDF
Sentiment Index of the Russian Speaking Facebook
Alexander Panchenko
 
PDF
Similarity Measures for Semantic Relation Extraction
Alexander Panchenko
 
PDF
Dmitry Gubanov. An Approach to the Study of Formal and Informal Relations of ...
Alexander Panchenko
 
PDF
Detecting Gender by Full Name: Experiments with the Russian Language
Alexander Panchenko
 
PDF
Document
Alexander Panchenko
 
PDF
Вычислительная лексическая семантика: метрики семантической близости и их при...
Alexander Panchenko
 
PDF
Semantic Similarity Measures for Semantic Relation Extraction
Alexander Panchenko
 
Building a Web-Scale Dependency-Parsed Corpus from Common Crawl
Alexander Panchenko
 
Improving Hypernymy Extraction with Distributional Semantic Classes
Alexander Panchenko
 
IIT-UHH at SemEval-2017 Task 3: Exploring Multiple Features for Community Que...
Alexander Panchenko
 
Fighting with Sparsity of the Synonymy Dictionaries for Automatic Synset Indu...
Alexander Panchenko
 
The 6th Conference on Analysis of Images, Social Networks, and Texts (AIST 2...
Alexander Panchenko
 
Using Linked Disambiguated Distributional Networks for Word Sense Disambiguation
Alexander Panchenko
 
Unsupervised Does Not Mean Uninterpretable: The Case for Word Sense Induction...
Alexander Panchenko
 
Getting started in Apache Spark and Flink (with Scala) - Part II
Alexander Panchenko
 
IIT-TUDA at SemEval-2016 Task 5: Beyond Sentiment Lexicon: Combining Domain ...
Alexander Panchenko
 
Text Analysis of Social Networks: Working with FB and VK Data
Alexander Panchenko
 
Неологизмы в социальной сети Фейсбук
Alexander Panchenko
 
Sentiment Index of the Russian Speaking Facebook
Alexander Panchenko
 
Similarity Measures for Semantic Relation Extraction
Alexander Panchenko
 
Dmitry Gubanov. An Approach to the Study of Formal and Informal Relations of ...
Alexander Panchenko
 
Detecting Gender by Full Name: Experiments with the Russian Language
Alexander Panchenko
 
Document
Alexander Panchenko
 
Вычислительная лексическая семантика: метрики семантической близости и их при...
Alexander Panchenko
 
Semantic Similarity Measures for Semantic Relation Extraction
Alexander Panchenko
 
Ad

Recently uploaded (20)

PPT
Chapter 52 introductory biology course Camp
abomajid13
 
PPTX
Models of Eucharyotic Chromosome Dr. Thirunahari Ugandhar.pptx
Head Department of Botany Govt Degree College Mahabubaba
 
PPTX
Chromosomal Aberrations Dr. Thirunahari Ugandhar.pptx
Head Department of Botany Govt Degree College Mahabubaba
 
PPTX
Introduction of Plant Ecology and Diversity Conservation
Sangam University, Bhilwara,Rajasthan
 
PDF
software engineering for computer science
shouryapratikofficia
 
PPTX
Antihypertensive Medicinal Chemistry Unit II BP501T.pptx
Dhanashri Dupade
 
PPTX
The Female Reproductive System - Grade 10 ppt
chieneecauso
 
PDF
SOCIAL PSYCHOLOGY_ CHAPTER 2.pdf- the self in a social world
coke pix
 
PPTX
complications of tooth extraction.pptx FIRM B.pptx
LytonChilembo
 
PPTX
ELS 2ND QUARTER 2 FOR HUMSS STUDENTS.pptx
FemaroseTelan
 
PPT
ZooLec Chapter 13 (Digestive System).ppt
normanquilantang1
 
PDF
SWAG Research Lab Scientific Publications
swagresearchlab
 
PPT
Chapter 6 Introductory course Biology Camp
abomajid13
 
PDF
Chemistry and Changes 8th Grade Science .pdf
lukefranke1
 
PDF
CHEM - GOC general organic chemistry.ppt
asma191141
 
PPTX
23ME402 Materials and Metallurgy- PPT.pptx
AnonymousmRBbdopMKf
 
PPT
what do you want to know about myeloprolifritive disorders .ppt
lasheseye676
 
PPTX
EPILEPSY UPDATE in kkm malaysia today new
luqarif119
 
PDF
Pharmacokinetics Lecture_Study Material.pdf
havan9956
 
PPTX
Bacterial and protozoal infections in pregnancy.pptx
aishuansari
 
Chapter 52 introductory biology course Camp
abomajid13
 
Models of Eucharyotic Chromosome Dr. Thirunahari Ugandhar.pptx
Head Department of Botany Govt Degree College Mahabubaba
 
Chromosomal Aberrations Dr. Thirunahari Ugandhar.pptx
Head Department of Botany Govt Degree College Mahabubaba
 
Introduction of Plant Ecology and Diversity Conservation
Sangam University, Bhilwara,Rajasthan
 
software engineering for computer science
shouryapratikofficia
 
Antihypertensive Medicinal Chemistry Unit II BP501T.pptx
Dhanashri Dupade
 
The Female Reproductive System - Grade 10 ppt
chieneecauso
 
SOCIAL PSYCHOLOGY_ CHAPTER 2.pdf- the self in a social world
coke pix
 
complications of tooth extraction.pptx FIRM B.pptx
LytonChilembo
 
ELS 2ND QUARTER 2 FOR HUMSS STUDENTS.pptx
FemaroseTelan
 
ZooLec Chapter 13 (Digestive System).ppt
normanquilantang1
 
SWAG Research Lab Scientific Publications
swagresearchlab
 
Chapter 6 Introductory course Biology Camp
abomajid13
 
Chemistry and Changes 8th Grade Science .pdf
lukefranke1
 
CHEM - GOC general organic chemistry.ppt
asma191141
 
23ME402 Materials and Metallurgy- PPT.pptx
AnonymousmRBbdopMKf
 
what do you want to know about myeloprolifritive disorders .ppt
lasheseye676
 
EPILEPSY UPDATE in kkm malaysia today new
luqarif119
 
Pharmacokinetics Lecture_Study Material.pdf
havan9956
 
Bacterial and protozoal infections in pregnancy.pptx
aishuansari
 

Graph's not dead: from unsupervised induction of linguistic structures from text towards applications in deep learning

  • 1. Alexander Panchenko From unsupervised induction of linguistic structures from text towards applications in deep learning
  • 2. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 2/98 In close collaboration with …
  • 3. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 3/98 Andrei Kutuzov Eugen Ruppert Fide Marten Nikolay Arefyev Steffen Remus Martin Riedl Hubert Naets Maria Pelevina Anastasiya Lopukhina Konstantin Lopukhin In collaboration with …
  • 4. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 4/98 Overview
  • 5. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 5/98 Inducing word sense representations: word sense embeddings via retrofitting [Pelevina et al., 2016, Remus & Biemann, 2018]; inducing synsets [Ustalov et al., 2017b, Ustalov et al., 2017a, Ustalov et al., 2018b] inducing semantic classes [Panchenko et al., 2018b] Overview Overview
  • 6. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 5/98 Inducing word sense representations: word sense embeddings via retrofitting [Pelevina et al., 2016, Remus & Biemann, 2018]; inducing synsets [Ustalov et al., 2017b, Ustalov et al., 2017a, Ustalov et al., 2018b] inducing semantic classes [Panchenko et al., 2018b] Making induced senses interpretable [Panchenko et al., 2017b, Panchenko et al., 2017c] Overview Overview
  • 7. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 5/98 Inducing word sense representations: word sense embeddings via retrofitting [Pelevina et al., 2016, Remus & Biemann, 2018]; inducing synsets [Ustalov et al., 2017b, Ustalov et al., 2017a, Ustalov et al., 2018b] inducing semantic classes [Panchenko et al., 2018b] Making induced senses interpretable [Panchenko et al., 2017b, Panchenko et al., 2017c] Linking induced word senses to lexical resources [Panchenko, 2016, Faralli et al., 2016, Panchenko et al., 2017a, Biemann et al., 2018] Overview Overview
  • 8. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 6/98 A shared task on word sense induction [Panchenko et al., 2018a, Arefyev et al., 2018] Overview Overview
  • 9. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 6/98 A shared task on word sense induction [Panchenko et al., 2018a, Arefyev et al., 2018] Inducing semantic frames [Ustalov et al., 2018a] Inducing FrameNet-like structures; …using multi-way clustering. Overview Overview
  • 10. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 6/98 A shared task on word sense induction [Panchenko et al., 2018a, Arefyev et al., 2018] Inducing semantic frames [Ustalov et al., 2018a] Inducing FrameNet-like structures; …using multi-way clustering. Learning graph/network embeddings [ongoing joint work with Andrei Kutuzov and Chris Biemann] How to represent induced networks/graphs? … so that they can be used in deep learning architectures. …effectively and efficiently. Overview Overview
  • 11. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 7/98 Inducing word sense representations
  • 12. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 8/98 Inducing word sense representations Word vs sense embeddings
  • 13. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 9/98 Inducing word sense representations Word vs sense embeddings
  • 14. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 10/98 Inducing word sense representations Related work
  • 15. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 11/98 AutoExtend [Rothe & Schütze, 2015] * image is reproduced from the original paper Inducing word sense representations Related work: knowledge-based
  • 16. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 12/98 Adagram [Bartunov et al., 2016] Multiple vector representations θ for each word: Inducing word sense representations Related work: knowledge-free
  • 17. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 12/98 Adagram [Bartunov et al., 2016] Multiple vector representations θ for each word: p(Y, Z, β|X, α, θ) = V∏ w=1 ∞∏ k=1 p(βwk|α) N∏ i=1 [p(zi|xi, β) C∏ j=1 p(yij|zi, xi, θ zi – a hidden variable: a sense index of word xi in context C; α – a meta-parameter controlling number of senses. Inducing word sense representations Related work: knowledge-free
  • 18. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 12/98 Adagram [Bartunov et al., 2016] Multiple vector representations θ for each word: p(Y, Z, β|X, α, θ) = V∏ w=1 ∞∏ k=1 p(βwk|α) N∏ i=1 [p(zi|xi, β) C∏ j=1 p(yij|zi, xi, θ zi – a hidden variable: a sense index of word xi in context C; α – a meta-parameter controlling number of senses. See also: [Neelakantan et al., 2014] and [Li and Jurafsky, 2015] Inducing word sense representations Related work: knowledge-free
  • 19. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 13/98 Word sense induction (WSI) based on graph clustering: [Lin, 1998] [Pantel and Lin, 2002] [Widdows and Dorow, 2002] Chinese Whispers [Biemann, 2006] [Hope and Keller, 2013] Inducing word sense representations Related work: word sense induction
  • 20. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 14/98 * source of the image: http://ic.pics.livejournal.com/blagin_anton/33716210/2701748/2701748_800.jpg Inducing word sense representations Related work: Chinese Whispers#1
  • 21. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 15/98 Inducing word sense representations Related work: Chinese Whispers#2
  • 22. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 16/98 Inducing word sense representations Related work: Chinese Whispers#2
  • 23. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 17/98 Inducing word sense representations Related work: Chinese Whispers#2
  • 24. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 18/98 RepL4NLP@ACL’16 [Pelevina et al., 2016], LREC’18 [Remus & Biemann, 2018] Prior methods: Induce inventory by clustering of word instances Use existing sense inventories Our method: Input: word embeddings Output: word sense embeddings Word sense induction by clustering of word ego-networks Inducing word sense representations Sense embeddings using retrofitting
  • 25. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 19/98 From word embeddings to sense embeddings Calculate Word Similarity Graph Learning Word Vectors Word Sense Induction Text Corpus Word Vectors Word Similarity Graph Pooling of Word Vectors Sense Inventory Sense Vectors 1 2 4 3 Inducing word sense representations Sense embeddings using retrofitting
  • 26. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 20/98 Word sense induction using ego-network clustering Inducing word sense representations Sense embeddings using retrofitting
  • 27. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 21/98 Neighbours of Word and Sense Vectors Vector Nearest Neighbors table tray, bottom, diagram, bucket, brackets, stack, basket, list, parenthesis, cup, saucer, pile, playfield, bracket, pot, drop-down, cue, plate Inducing word sense representations Sense embeddings using retrofitting
  • 28. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 21/98 Neighbours of Word and Sense Vectors Vector Nearest Neighbors table tray, bottom, diagram, bucket, brackets, stack, basket, list, parenthesis, cup, saucer, pile, playfield, bracket, pot, drop-down, cue, plate table#0 leftmost#0, column#1, tableau#1, indent#1, bracket#3, pointer#0, footer#1, cursor#1, diagram#0, grid#0 table#1 pile#1, stool#1, tray#0, basket#0, bowl#1, bucket#0, box#0, cage#0, saucer#3, mirror#1, pan#1, lid#0 Inducing word sense representations Sense embeddings using retrofitting
  • 29. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 22/98 Word and sense embeddings of words iron and vitamin. LREC’18 [Remus & Biemann, 2018] Inducing word sense representations Sense embeddings using retrofitting
  • 30. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 23/98 Word Sense Disambiguation 1 Context extraction: use context words around the target word 2 Context filtering: based on context word’s relevance for disambiguation 3 Sense choice in context: maximise similarity between a context vector and a sense vector Inducing word sense representations Sense embeddings using retrofitting
  • 31. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 24/98 Inducing word sense representations Sense embeddings using retrofitting
  • 32. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 25/98 Inducing word sense representations Sense embeddings using retrofitting
  • 33. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 26/98 Inducing word sense representations Sense embeddings using retrofitting
  • 34. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 27/98 Inducing word sense representations Sense embeddings using retrofitting
  • 35. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 28/98 Unsupervised WSD SemEval’13, ReprL4NLP [Pelevina et al., 2016]: Model Jacc. Tau WNDCG F.NMI F.B-Cubed AI-KU (add1000) 0.176 0.609 0.205 0.033 0.317 AI-KU 0.176 0.619 0.393 0.066 0.382 AI-KU (remove5-add1000) 0.228 0.654 0.330 0.040 0.463 Unimelb (5p) 0.198 0.623 0.374 0.056 0.475 Unimelb (50k) 0.198 0.633 0.384 0.060 0.494 UoS (#WN senses) 0.171 0.600 0.298 0.046 0.186 UoS (top-3) 0.220 0.637 0.370 0.044 0.451 La Sapienza (1) 0.131 0.544 0.332 – – La Sapienza (2) 0.131 0.535 0.394 – – AdaGram, α = 0.05, 100 dim 0.274 0.644 0.318 0.058 0.470 w2v 0.197 0.615 0.291 0.011 0.615 w2v (nouns) 0.179 0.626 0.304 0.011 0.623 JBT 0.205 0.624 0.291 0.017 0.598 JBT (nouns) 0.198 0.643 0.310 0.031 0.595 TWSI (nouns) 0.215 0.651 0.318 0.030 0.573 Inducing word sense representations Sense embeddings using retrofitting
  • 36. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 29/98 Semantic relatedness, LREC’2018 [Remus & Biemann, 2018]: autoextend adagram SGNS . glove . sympat . LSAbow . LSAhal . paragramSL . SimLex999 0.45 0.29 0.44 0.37 0.54 0.30 0.27 0.68 MEN 0.72 0.67 0.77 0.73 0.53 0.67 0.71 0.77 SimVerb 0.43 0.27 0.36 0.23 0.37 0.15 0.19 0.53 WordSim353 0.58 0.61 0.70 0.61 0.47 0.67 0.59 0.72 SimLex999-N 0.44 0.33 0.45 0.39 0.48 0.32 0.34 0.68 MEN-N 0.72 0.68 0.77 ____ 0.76 ____ 0.57 ____ 0.71 ____ 0.73 ____ 0.78 ____ Inducing word sense representations Sense embeddings using retrofitting
  • 37. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 30/98 Unsupervised WSD SemEval’13, ReprL4NLP [Pelevina et al., 2016]: comparable to SOTA, incl. sense embeddings. Semantic relatedness, LREC’2018 [Remus & Biemann, 2018]: autoextend adagram SGNS SGNS+senses glove glove+senses sympat sympat+senses LSAbow LSAbow+senses LSAhal LSAhal+senses paragramSL paragramSL+senses SimLex999 0.45 0.29 0.44 0.46 0.37 0.41 0.54 0.55 0.30 0.39 0.27 0.38 0.68 0.64 MEN 0.72 0.67 0.77 0.78 0.73 0.77 0.53 0.68 0.67 0.70 0.71 0.74 0.77 0.80 SimVerb 0.43 0.27 0.36 0.39 0.23 0.30 0.37 0.45 0.15 0.22 0.19 0.28 0.53 0.53 WordSim353 0.58 0.61 0.70 0.69 0.61 0.65 0.47 0.62 0.67 0.66 0.59 0.63 0.72 0.73 SimLex999-N 0.44 0.33 0.45 0.50 0.39 0.47 0.48 0.55 0.32 0.46 0.34 0.44 0.68 0.66 MEN-N 0.72 0.68 0.77 0.79 0.76 0.80 0.57 0.74 0.71 0.73 0.73 0.76 0.78 0.81 Inducing word sense representations Sense embeddings using retrofitting
  • 38. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 31/98 ACL’17 [Ustalov et al., 2017b] Examples of extracted synsets: Size Synset 2 {decimal point, dot} 3 {gullet, throat, food pipe} 4 {microwave meal, ready meal, TV dinner, frozen dinner} 5 {objective case, accusative case, oblique case, object case, accusative} 6 {radiotheater, dramatizedaudiobook, audiotheater, ra- dio play, radio drama, audio play} Inducing word sense representations Synset induction
  • 39. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 32/98 Outline of the ’Watset’ method: Background Corpus Synonymy Dictionary Learning Word Embeddings Graph Construction Synsets Word Similarities Ambiguous Weighted Graph Local Clustering: Word Sense Induction Global Clustering: Synset Induction Sense Inventory Disambiguation of Neighbors Disambiguated Weighted Graph Local-Global Fuzzy Graph Clustering Inducing word sense representations Synset induction
  • 40. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 33/98 Inducing word sense representations Synset induction
  • 41. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 34/98 Inducing word sense representations Synset induction
  • 42. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 35/98 Inducing word sense representations Synset induction
  • 43. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 36/98 CW MCL MaxMax ECO CPM Watset 0.0 0.1 0.2 0.3 WordNet (English) F−score CW MCL MaxMax ECO CPM Watset 0.0 0.1 0.2 0.3 BabelNet (English) F−score Inducing word sense representations Synset induction
  • 44. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 36/98 CW MCL MaxMax ECO CPM Watset 0.0 0.1 0.2 0.3 WordNet (English) F−score CW MCL MaxMax ECO CPM Watset 0.0 0.1 0.2 0.3 BabelNet (English) F−score CW MCL MaxMax ECO CPM Watset 0.00 0.05 0.10 0.15 0.20 RuWordNet (Russian) F−score CW MCL MaxMax ECO CPM Watset 0.0 0.1 0.2 0.3 0.4 YARN (Russian) F−score Inducing word sense representations Synset induction
  • 45. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 37/98 Word Sense Local Sense Cluster: Related Senses Hypernyms mango#0 peach#1, grape#0, plum#0, apple#0, apricot#0, watermelon#1, banana#1, coconut#0, pear#0, fig#0, melon#0, mangosteen#0, … fruit#0, food#0, … apple#0 mango#0, pineapple#0, banana#1, melon#0, grape#0, peach#1, watermelon#1, apricot#0, cranberry#0, pumpkin#0, mangosteen#0, … fruit#0, crop#0, … Java#1 C#4, Python#3, Apache#3, Ruby#6, Flash#1, C++#0, SQL#0, ASP#2, Visual Basic#1, CSS#0, Delphi#2, MySQL#0, Excel#0, Pascal#0, … programming language#3, lan- guage#0, … Python#3 PHP#0, Pascal#0, Java#1, SQL#0, Visual Ba- sic#1, C++#0, JavaScript#0, Apache#3, Haskell#5, .NET#1, C#4, SQL Server#0, … language#0, tech- nology#0, … Inducing word sense representations Sample of induced sense inventory
  • 46. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 38/98 ID Global Sense Cluster: Semantic Class Hypernyms 1 peach#1, banana#1, pineapple#0, berry#0, black- berry#0, grapefruit#0, strawberry#0, blueberry#0, mango#0, grape#0, melon#0, orange#0, pear#0, plum#0, raspberry#0, watermelon#0, apple#0, apri- cot#0, watermelon#0, pumpkin#0, berry#0, man- gosteen#0, … vegetable#0, fruit#0, crop#0, ingredi- ent#0, food#0, · 2 C#4, Basic#2, Haskell#5, Flash#1, Java#1, Pas- cal#0, Ruby#6, PHP#0, Ada#1, Oracle#3, Python#3, Apache#3, Visual Basic#1, ASP#2, Delphi#2, SQL Server#0, CSS#0, AJAX#0, JavaScript#0, SQL Server#0, Apache#3, Delphi#2, Haskell#5, .NET#1, CSS#0, … programming lan- guage#3, technol- ogy#0, language#0, format#2, app#0 Inducing word sense representations Sample of induced semantic classes
  • 47. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 39/98 Text Corpus Representing Senses with Ego Networks Semantic Classes Word Sense Induction from Text Corpus Sense Graph Construction Clustering of Word Senes Labeling Sense Clusters with Hypernyms Induced Word Senses Sense Ego-Networks Global Sense Graph s Noisy Hypernyms Cleansed Hypernyms Induction of Semantic Classes Global Sense Clusters Inducing word sense representations Induction of semantic classes
  • 48. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 40/98 Filtering noisy hypernyms with semantic classes LREC’18 [Panchenko et al., 2018b]: fruit#1 food#0 apple#2 mango#0 pear#0 Hypernyms, Sense Cluster, mangosteen#0 city#2 Removed Wrong Added Missing Inducing word sense representations Induction of sense semantic classes
  • 49. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 41/98 http://panchenko.me/data/joint/nodes20000-layers7 Inducing word sense representations Global sense clustering
  • 50. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 42/98 Inducing word sense representations Global sense clustering
  • 51. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 43/98 Filtering of a noisy hypernymy database with semantic classes. LREC’18 [Panchenko et al., 2018b] Precision Recall F-score Original Hypernyms (Seitner et al., 2016) 0.475 0.546 0.508 Semantic Classes (coarse-grained) 0.541 0.679 0.602 Inducing word sense representations Induction of sense semantic classes
  • 52. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 44/98 Making induced senses interpretable
  • 53. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 45/98 Knowledge-based sense representations are interpretable Making induced senses interpretable Making induced senses interpretable
  • 54. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 46/98 Most knowledge-free sense representations are uninterpretable Making induced senses interpretable Making induced senses interpretable
  • 55. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 47/98 Making induced senses interpretable Making induced senses interpretable
  • 56. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 48/98 Hypernymy prediction in context. EMNLP’17 [Panchenko et al., 2017b] Making induced senses interpretable Making induced senses interpretable
  • 57. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 49/98 11.702 sentences, 863 words with avg.polysemy of 3.1. WSD Model Accuracy Inventory Features Hypers HyperHypers Word Senses Random 0.257 0.610 Word Senses MFS 0.292 0.682 Word Senses Cluster Words 0.291 0.650 Word Senses Context Words 0.308 0.686 Making induced senses interpretable Making induced senses interpretable
  • 58. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 49/98 11.702 sentences, 863 words with avg.polysemy of 3.1. WSD Model Accuracy Inventory Features Hypers HyperHypers Word Senses Random 0.257 0.610 Word Senses MFS 0.292 0.682 Word Senses Cluster Words 0.291 0.650 Word Senses Context Words 0.308 0.686 Super Senses Random 0.001 0.001 Super Senses MFS 0.001 0.001 Super Senses Cluster Words 0.174 0.365 Super Senses Context Words 0.086 0.188 Making induced senses interpretable Making induced senses interpretable
  • 59. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 50/98 Linking induced senses to resources
  • 60. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 51/98 Text Corpus Linking Induced Senses to Senses of the LR Induce a Graph of Sem. Related Words Enriched Lexical Resource Graph of Related Words Word Sense Induction Labeling Senses with Hypernyms Disambiguation of Neighbours Typing of the Unmapped Induced Senses Word Sense Inventory Labeled Word Senses PCZ Construction of Proto-Conceptualization (PCZ) Linking Proto-Conceptualization to Lexical Resource Part. Linked Senses to the LR Lexical Resource (LR): WordNet, BabelNet, ... Construction of sense feature representations Graph of Related Senses LREC’16 [Panchenko, 2016], ISWC’16 [Faralli et al., 2016], SENSE@EACL’17 [Panchenko et al., 2017a], NLE’18 [Biemann et al., 2018] Linking induced senses to resources Linking induced senses to resources
  • 61. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 52/98 Word AdaGram BabelNet AdaGram BoW BabelNet BoW python 2 bn:01713224n perl, php, java, smalltalk, ruby, lua, tcl, scripting, javascript, bindings, binding, programming, coldfusion, actionscript, net, . . . language, programming, python- ista, python programming, python3, python2, level, com- puter, pythonistas, python3000, python 1 bn:01157670n monty, circus, spamalot, python, magoo, muppet, snoopy, fea- turette, disney, tunes, tune, clas- sic, shorts, short, apocalypse, . . . monty, comedy, monty python, british, monte, monte python, troupe, pythonesque, foot, artist, record, surreal, terry, . . . python 3 bn:00046456n spectacled, unicornis, snake, gi- ant, caiman, leopard, squirrel, crocodile, horned, cat, mole, ele- phant, opossum, pheasant, . . . molurus, indian, boa, tigris, tiger python, rock, tiger, indian python, reptile, python molurus, indian rock python, coluber, . . . python 4 bn:01157670n circus, fly, flying, dusk, lizard, moth, unicorn, puff, adder, vul- ture, tyrannosaurus, zephyr, bad- ger, . . . monty, comedy, monty python, british, monte, monte python, troupe, pythonesque, foot, artist, record, surreal, terry, . . . python 1 bn:00473212n monty, circus, spamalot, python, magoo, muppet, snoopy, fea- turette, disney, tunes, tune, clas- sic, shorts, short, apocalypse, . . . pictures, monty, python monty pictures, limited, company, python pictures limited, king- dom, picture, serve, director, . . . python 1 bn:03489893n monty, circus, spamalot, python, magoo, muppet, snoopy, fea- turette, disney, tunes, tune, clas- sic, shorts, short, apocalypse, . . . film, horror, movie, clabaugh, richard, monster, century, direct, snake, python movie, television, giant, natural, language, for-tv, . . . Linking induced senses to resources Linking induced senses to resources
  • 62. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 53/98 Model Representation of the Sense ”disk (medium)” WordNet memory, device, floppy, disk, hard, disk, disk, computer, science, computing, diskette, fixed, disk, floppy, magnetic, disc, magnetic, disk, hard, disc, storage, device WordNet + Linked recorder, disk, floppy, console, diskette, handset, desktop, iPhone, iPod, HDTV, kit, RAM, Discs, Blu- ray, computer, GB, microchip, site, cartridge, printer, tv, VCR, Disc, player, LCD, software, component, camcorder, cellphone, card, monitor, display, burner, Web, stereo, internet, model, iTunes, turntable, chip, cable, camera, iphone, notebook, device, server, surface, wafer, page, drive, laptop, screen, pc, television, hardware, YouTube, dvr, DVD, product, folder, VCR, radio, phone, circuitry, partition, megabyte, peripheral, format, machine, tuner, website, merchandise, equipment, gb, discs, MP3, hard-drive, piece, video, storage device, memory device, microphone, hd, EP, content, soundtrack, webcam, system, blade, graphic, microprocessor, collection, document, programming, battery, key- board, HD, handheld, CDs, reel, web, material, hard-disk, ep, chart, debut, configuration, recording, album, broadcast, download, fixed disk, planet, pda, microfilm, iPod, videotape, text, cylinder, cpu, canvas, label, sampler, workstation, electrode, magnetic disc, catheter, magnetic disk, Video, mo- bile, cd, song, modem, mouse, tube, set, ipad, signal, substrate, vinyl, music, clip, pad, audio, com- pilation, memory, message, reissue, ram, CD, subsystem, hdd, touchscreen, electronics, demo, shell, sensor, file, shelf, processor, cassette, extra, mainframe, motherboard, floppy disk, lp, tape, version, kilobyte, pacemaker, browser, Playstation, pager, module, cache, DVD, movie, Windows, cd-rom, e- book, valve, directory, harddrive, smartphone, audiotape, technology, hard disk, show, computing, computer science, Blu-Ray, blu-ray, HDD, HD-DVD, scanner, hard disc, gadget, booklet, copier, play- back, TiVo, controller, filter, DVDs, gigabyte, paper, mp3, CPU, dvd-r, pipe, cd-r, playlist, slot, VHS, film, videocassette, interface, adapter, database, manual, book, channel, changer, storage Linking induced senses to resources Linking induced senses to resources
  • 63. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 54/98 Evaluation of linking accuracy: Linking induced senses to resources Linking induced senses to resources
  • 64. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 55/98 Evaluation of enriched representations based on WSD: Linking induced senses to resources Linking induced senses to resources
  • 65. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 56/98 Shared task on word sense induction
  • 66. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 57/98 An ACL SIGSLAV sponsored shared task on word sense induction (WSI) for the Russian language. More details: https://russe.nlpub.org/2018/wsi Shared task on word sense induction A shared task on WSI
  • 67. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 58/98 Target word, e.g. “bank”. Shared task on word sense induction A lexical sample WSI task
  • 68. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 58/98 Target word, e.g. “bank”. Contexts where the word occurs, e.g.: “river bank is a slope beside a body of water” “bank is a financial institution that accepts deposits” “Oh, the bank was robbed. They took about a million dollars.” “bank of Elbe is a good and popular hangout spot complete with good food and fun” Shared task on word sense induction A lexical sample WSI task
  • 69. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 58/98 Target word, e.g. “bank”. Contexts where the word occurs, e.g.: “river bank is a slope beside a body of water” “bank is a financial institution that accepts deposits” “Oh, the bank was robbed. They took about a million dollars.” “bank of Elbe is a good and popular hangout spot complete with good food and fun” You need to group the contexts by senses: “river bank is a slope beside a body of water” “bank of Elbe is a good and popular hangout spot complete with good food and fun” “bank is a financial institution that accepts deposits” “Oh, the bank was robbed. They took about a million dollars.” Shared task on word sense induction A lexical sample WSI task
  • 70. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 59/98 Shared task on word sense induction Dataset based on Wikipedia
  • 71. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 60/98 Shared task on word sense induction Dataset based on RNC
  • 72. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 61/98 Shared task on word sense induction Dataset based on dictionary glosses
  • 73. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 62/98 Shared task on word sense induction A sample from the wiki-wiki dataset
  • 74. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 63/98 Shared task on word sense induction A sample from the wiki-wiki dataset
  • 75. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 64/98 Shared task on word sense induction A sample from the wiki-wiki dataset
  • 76. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 65/98 Shared task on word sense induction A sample from the bts-rnc dataset
  • 77. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 66/98 Shared task on word sense induction A sample from the active-dict dataset
  • 78. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 67/98 1 Get the neighbors of a target word, e.g. “bank”: 1 lender 2 river 3 citybank 4 slope 5 … 2 Get similar to “bank” and dissimilar to “lender”: 1 river 2 slope 3 land 4 … 3 Compute distances to “lender” and “river”. Shared task on word sense induction jamsic: sense induction
  • 79. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 68/98 Induction of semantic frames
  • 80. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 69/98 Induction of semantic frames FrameNet: frame “Kidnapping”
  • 81. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 70/98 ACL’2018 [Ustalov et al., 2018a] Example of a LU tricluster corresponding to the “Kidnapping” frame from FrameNet. FrameNet Role Lexical Units (LU) Perpetrator Subject kidnapper, alien, militant FEE Verb snatch, kidnap, abduct Victim Object son, people, soldier, child Induction of semantic frames Frame induction as a triclustering
  • 82. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 71/98 Induction of semantic frames SVO triple elements
  • 83. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 72/98 Officer|chair|Committee officer|head|team mayor|lead|city officer|lead|company Mayor|lead|city boss|lead|company chairman|lead|company director|lead|department chief|lead|department president|lead|government president|lead|state director|lead|company president|lead|department officer|chair|committee Chief|lead|department chairman|lead|committee Director|lead|Department Director|lead|department Director|lead|agency Director|lead|company minister|lead|team Director|head|team director|head|team Chairman|lead|company Chairman|lead|Committee President|lead|company Director|chair|Committee President|lead|party President|head|teamleader|head|team Director|chair|committee director|chair|committee Director|head|Department president|head|team director|head|department director|head|agency director|head|committee Chairman|run|committee Chairman|chair|Committee Chairman|chair|committee President|chair|Committee President|chair|committee Governor|lead|state chairman|head|committee chairman|run|committee president|chair|committee president|head|committee president|chair|Committee Minister|chair|committee representative|chair|committee representative|head|committee General|command|department General|command|Department General|head|Department General|head|department officer|head|department minister|head|department leader|head|agency leader|head|party leader|head|committee leader|head|department minister|head|committee King|run|company leader|head|government Minister|head|government president|head|government Officer|chair|Committee officer|head|team mayor|lead|city officer|lead|company Mayor|lead|city boss|lead|company chairman|lead|company director|lead|department chief|lead|department president|lead|government president|lead|state director|lead|company president|lead|department officer|chair|committee Chief|lead|department chairman|lead|committee Director|lead|Department Director|lead|department Director|lead|agency Director|lead|company minister|lead|team Director|head|team director|head|team Chairman|lead|company Chairman|lead|Committee President|lead|company Director|chair|Committee President|lead|party President|head|teamleader|head|team Director|chair|committee director|chair|committee Director|head|Department president|head|team director|head|department director|head|agency director|head|committee Chairman|run|committee Chairman|chair|Committee Chairman|chair|committee President|chair|Committee President|chair|committee Governor|lead|state chairman|head|committee chairman|run|committee president|chair|committee president|head|committee president|chair|Committee Minister|chair|committee representative|chair|committee representative|head|committee General|command|department General|command|Department General|head|Department General|head|department officer|head|department minister|head|department leader|head|agency leader|head|party leader|head|committee leader|head|department minister|head|committee King|run|company leader|head|government Minister|head|government president|head|government Induction of semantic frames An SVO triple graph
  • 84. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 73/98 Input: an embedding model v ∈ V → ⃗v ∈ Rd, a set of SVO triples T ⊆ V 3, the number of nearest neighbors k ∈ N, a graph clustering algorithm Cluster. Induction of semantic frames Triframes frame induction
  • 85. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 73/98 Input: an embedding model v ∈ V → ⃗v ∈ Rd, a set of SVO triples T ⊆ V 3, the number of nearest neighbors k ∈ N, a graph clustering algorithm Cluster. Output: a set of triframes F. Induction of semantic frames Triframes frame induction
  • 86. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 73/98 Input: an embedding model v ∈ V → ⃗v ∈ Rd, a set of SVO triples T ⊆ V 3, the number of nearest neighbors k ∈ N, a graph clustering algorithm Cluster. Output: a set of triframes F. S ← {t→ ⃗t ∈ R3d : t ∈ T} E ← {(t, t′) ∈ T2 : t′ ∈ NNS k (⃗t), t ̸= t′} F ← ∅ for all C ∈ Cluster(T, E) do fs ← {s ∈ V : (s, v, o) ∈ C} fv ← {v ∈ V : (s, v, o) ∈ C} fo ← {o ∈ V : (s, v, o) ∈ C} F ← F ∪ {(fs, fv, fo)} return F Induction of semantic frames Triframes frame induction
  • 87. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 74/98 Frame # 848 Subjects: Company, firm, company Verbs: buy, supply, discharge, purchase, expect Objects: book, supply, house, land, share, company, grain, which, item, product, ticket, work, this, equipment, House, it, film, water, something, she, what, service, plant, time Induction of semantic frames Example of an extracted frame
  • 88. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 75/98 Frame # 849 Subjects: student, scientist, we, pupil, member, company, man, nobody, you, they, US, group, it, people, Man, user, he Verbs: do, test, perform, execute, conduct Objects: experiment, test Induction of semantic frames Example of an extracted frame
  • 89. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 76/98 Frame # 3207 Subjects: people, we, they, you Verbs: feel, seek, look, search Objects: housing, inspiration, gold, witness, part- ner, accommodation, Partner Induction of semantic frames Example of an extracted frame
  • 90. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 77/98 Dataset # instances # unique # clusters FrameNet Triples 99,744 94,170 383 Poly. Verb Classes 246 110 62 Induction of semantic frames Evaluation datasets
  • 91. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 78/98 Dataset # instances # unique # clusters FrameNet Triples 99,744 94,170 383 Poly. Verb Classes 246 110 62 Induction of semantic frames Evaluation settings
  • 92. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 78/98 Dataset # instances # unique # clusters FrameNet Triples 99,744 94,170 383 Poly. Verb Classes 246 110 62 Quality Measures: nmPU: normalized modified purity, niPU: normalized inverse purity. Induction of semantic frames Evaluation settings
  • 93. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 79/98 F1-scores for verbs, subjects, objects, frames Induction of semantic frames Results: comparison to state-of-art
  • 94. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 80/98 Graph embeddings
  • 95. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 81/98 Graph embeddings Text: sparse symbolic representation
  • 96. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 81/98 Image source: https://www.tensorflow.org/tutorials/word2vec Graph embeddings Text: sparse symbolic representation
  • 97. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 82/98 Graph embeddings Graph: sparse symbolic representation
  • 98. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 83/98 From a survey on graph embeddings [Hamilton et al., 2017]: Graph embeddings Embedding graph into a vector space
  • 99. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 84/98 From a survey on graph embeddings [Hamilton et al., 2017]: Graph embeddings Learning with an “autoencoder”
  • 100. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 85/98 From a survey on graph embeddings [Hamilton et al., 2017]: Graph embeddings Some established approaches
  • 101. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 86/98 An submitted joint work with Andrei Kutuzov and Chris Biemann: Given a tree (V, E) Leackock-Chodorow (LCH) similarity measure: sim(vi, vj) = − log shortest_path_distance(vi, vj) 2h Graph embeddings Graph embeddings using similarities
  • 102. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 86/98 An submitted joint work with Andrei Kutuzov and Chris Biemann: Given a tree (V, E) Leackock-Chodorow (LCH) similarity measure: sim(vi, vj) = − log shortest_path_distance(vi, vj) 2h Jiang-Conrath (JCN) similarity measure: sim(vi, vj) = 2 ln Plcs(vi, vj) ln P(vi) + ln P(vj) Graph embeddings Graph embeddings using similarities
  • 103. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 87/98 path2vec: Approximating Structural Node Similarities with Node Embeddings: J = 1 |T| ∑ (vi,vj)∈T (vi · vj − sim(vi, vj))2 , where: sim(vi, vj) - the value of a ‘gold’ similarity measure between a pair of nodes vi and vj; vi - an embeddings of node; T - training batch. Graph embeddings Graph embeddings using similarities
  • 104. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 88/98 Computation of 82,115 pairwise similarities: Model Running time LCH in NLTK 30 sec. JCN in NLTK 6.7 sec. FSE embeddings 0.713 sec. path2vec and other float vectors 0.007 sec. Graph embeddings Speedup: graph vs embeddings
  • 105. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 89/98 Spearman correlation scores with WordNet similarities on SimLex999 noun pairs: Selection of synsets Model JCN-SemCor JCN-Brown LCH WordNet 1.0 1.0 1.0 Node2vec 0.655 0.671 0.724 Deepwalk 0.775 0.774 0.868 FSE 0.830 0.820 0.900 path2vec 0.917 0.914 0.934 Graph embeddings Results: goodness of fit
  • 106. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 90/98 Spearman correlations with human SimLex999 noun similarities: Model Correlation Raw WordNet JCN-SemCor 0.487 Raw WordNet JCN-Brown 0.495 Raw WordNet LCH 0.513 node2vec [Grover & Leskovec, 2016] 0.450 Deepwalk [Perozzi et al., 2014] 0.533 FSE [Subercaze et al., 2015] 0.556 path2vec JCN-SemCor 0.526 path2vec JCN-Brown 0.487 path2vec LCH 0.522 Graph embeddings Results: SimLex999 dataset
  • 107. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 91/98 JCN (left) and LCH (right): Graph embeddings Results: SimLex999 dataset
  • 108. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 92/98 WSD: each column lists all the possible synsets for the corresponding word. Graph embeddings Results: word sense disambiguation
  • 109. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 93/98 F1 scores on Senseval-2 word sense disambiguation task: Model F-measure WordNet JCN-SemCor 0.620 WordNet JCN-Brown 0.561 WordNet LCH 0.547 node2vec [Grover & Leskovec, 2016] 0.501 Deepwalk [Perozzi et al., 2014] 0.528 FSE [Subercaze et al., 2015] 0.536 path2vec Batch size: 20 50 100 JCN-SemCor 0.543 0.543 0.535 JCN-Brown 0.538 0.515 0.542 LCH 0.540 0.535 0.536 Graph embeddings Results: word sense disambiguation
  • 110. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 94/98 Conclusion
  • 111. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 95/98 Conclusion Vectors + Graphs = ♡
  • 112. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 96/98 We can induce word senses, synsets, semantic classes, and semantic frames in a knowledge-free way using graph clustering and distributional models. Conclusion Take home messages
  • 113. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 96/98 We can induce word senses, synsets, semantic classes, and semantic frames in a knowledge-free way using graph clustering and distributional models. We can make the induced word senses interpretable in a knowledge-free way with hypernyms, images, definitions. Conclusion Take home messages
  • 114. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 96/98 We can induce word senses, synsets, semantic classes, and semantic frames in a knowledge-free way using graph clustering and distributional models. We can make the induced word senses interpretable in a knowledge-free way with hypernyms, images, definitions. We can link induced senses to lexical resources to improve performance of WSD; enrich lexical resources with emerging senses. Conclusion Take home messages
  • 115. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 96/98 We can induce word senses, synsets, semantic classes, and semantic frames in a knowledge-free way using graph clustering and distributional models. We can make the induced word senses interpretable in a knowledge-free way with hypernyms, images, definitions. We can link induced senses to lexical resources to improve performance of WSD; enrich lexical resources with emerging senses. We can represent language graphs using graph embeddings in deep neural models. Conclusion Take home messages
  • 116. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 97/98 A special issue on informing neural architectures for NLP with linguistic and background knowledge. … with Ivan Vulić and Simone Paolo Ponzetto. goo.gl/A76NGX Conclusion Natural Language Engineering journal
  • 117. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 Conclusion Thank you! Questions?
  • 118. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 Arefyev, N., Ermolaev, P., & Panchenko, A. (2018). How much does a word weigh? weighting word embeddings for word sense induction. arXiv preprint arXiv:1805.09209. Bartunov, S., Kondrashkin, D., Osokin, A., & Vetrov, D. (2016). Breaking sticks and ambiguities with adaptive skip-gram. In Artificial Intelligence and Statistics (pp. 130–138). Biemann, C., Faralli, S., Panchenko, A., & Ponzetto, S. P. (2018). A framework for enriching lexical semantic resources with distributional semantics. In Journal of Natural Language Engineering (pp. 56–64).: Cambridge Press. Faralli, S., Panchenko, A., Biemann, C., & Ponzetto, S. P. (2016). Linked disambiguated distributional semantic networks. In International Semantic Web Conference (pp. 56–64).: Springer. Grover, A. & Leskovec, J. (2016).
  • 119. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 Node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 855–864).: ACM. Hamilton, W. L., Ying, R., & Leskovec, J. (2017). Representation learning on graphs: Methods and applications. IEEE Data Engineering Bulletin, September 2017. Panchenko, A. (2016). Best of both worlds: Making word sense embeddings interpretable. In LREC. Panchenko, A., Faralli, S., Ponzetto, S. P., & Biemann, C. (2017a). Using linked disambiguated distributional networks for word sense disambiguation.
  • 120. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 In Proceedings of the 1st Workshop on Sense, Concept and Entity Representations and their Applications (pp. 72–78). Valencia, Spain: Association for Computational Linguistics. Panchenko, A., Lopukhina, A., Ustalov, D., Lopukhin, K., Arefyev, N., Leontyev, A., & Loukachevitch, N. (2018a). Russe’2018: A shared task on word sense induction for the russian language. arXiv preprint arXiv:1803.05795. Panchenko, A., Marten, F., Ruppert, E., Faralli, S., Ustalov, D., Ponzetto, S. P., & Biemann, C. (2017b). Unsupervised, knowledge-free, and interpretable word sense disambiguation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (pp. 91–96). Copenhagen, Denmark: Association for Computational Linguistics. Panchenko, A., Ruppert, E., Faralli, S., Ponzetto, S. P., & Biemann, C. (2017c).
  • 121. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 Unsupervised does not mean uninterpretable: The case for word sense induction and disambiguation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers (pp. 86–98). Valencia, Spain: Association for Computational Linguistics. Panchenko, A., Ustalov, D., Faralli, S., Ponzetto, S. P., & Biemann, C. (2018b). Improving hypernymy extraction with distributional semantic classes. In Proceedings of the LREC 2018 Miyazaki, Japan: European Language Resources Association. Pelevina, M., Arefiev, N., Biemann, C., & Panchenko, A. (2016). Making sense of word embeddings. In Proceedings of the 1st Workshop on Representation Learning for NLP (pp. 174–183). Berlin, Germany: Association for Computational Linguistics. Perozzi, B., Al-Rfou, R., & Skiena, S. (2014).
  • 122. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 701–710).: ACM. Remus, S. & Biemann, C. (2018). Retrofittingword representations for unsupervised sense aware word similarities. In Proceedings of the LREC 2018 Miyazaki, Japan: European Language Resources Association. Rothe, S. & Schütze, H. (2015). Autoextend: Extending word embeddings to embeddings for synsets and lexemes. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 1793–1803). Beijing, China: Association for Computational Linguistics. Subercaze, J., Gravier, C., & Laforest, F. (2015).
  • 123. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 On metric embedding for boosting semantic similarity computations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (pp. 8–14).: Association for Computational Linguistics. Ustalov, D., Chernoskutov, M., Biemann, C., & Panchenko, A. (2017a). Fighting with the sparsity of synonymy dictionaries for automatic synset induction. In International Conference on Analysis of Images, Social Networks and Texts (pp. 94–105).: Springer. Ustalov, D., Panchenko, A., & Biemann, C. (2017b). Watset: Automatic induction of synsets from a graph of synonyms. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp.
  • 124. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98 1579–1590). Vancouver, Canada: Association for Computational Linguistics. Ustalov, D., Panchenko, A., Kutuzov, A., Biemann, C., & Ponzetto, S. P. (2018a). Unsupervised semantic frame induction using triclustering. arXiv preprint arXiv:1805.04715. Ustalov, D., Teslenko, D., Panchenko, A., Chernoskutov, M., & Biemann, C. (2018b). Word sense disambiguation based on automatically induced synsets. In LREC 2018, 11th International Conference on Language Resources and Evaluation : 7-12 May 2018, Miyazaki (Japan) (pp. tba). Paris: European Language Resources Association, ELRA-ELDA. Accepted for publication.