Statistical Semantic入門 ~分布仮説からword2vecまで~
221 likes87,430 views
1. The document discusses various statistical and neural network-based models for representing words and modeling semantics, including LSI, PLSI, LDA, word2vec, and neural network language models. 2. These models represent words based on their distributional properties and contexts using techniques like matrix factorization, probabilistic modeling, and neural networks to learn vector representations. 3. Recent models like word2vec use neural networks to learn word embeddings that capture linguistic regularities and can be used for tasks like analogy-making and machine translation.
![[Bird+10]
10
10.1
10.2
10.3
10.4
10.5
10.6
10.7
10.8](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-5-320.jpg)
![[
+96]
5.
5.1
5.2
5.3
5.4](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-6-320.jpg)
![[Evert10]
NAACL2010 Stefan Evert
Semantic Models
Distributional](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-10-320.jpg)
![???
[Evert10]](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-11-320.jpg)
![???
2 cat
pig
knife
[Evert10]](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-12-320.jpg)
![dog
[Evert10]](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-13-320.jpg)
![[
13]
!
!
!](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-17-320.jpg)
![: Latent Semantic Indexing (LSI),
Latent Semantic Analysis (LSA) [Deerwester+90]
!
!
!
!](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-21-320.jpg)
![Probabilistic Latent Semantic
Indexing (PLSI) [Hofmann99]
!
LSI
!
!
!
ex:
LSI](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-29-320.jpg)
![Latent Dirichlet Allocation (LDA) [Blei03]
PLSI
! PLSI
LDA
!](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-31-320.jpg)
![Non-negative Matrix Factorization (NMF) [Lee
+99]
!
SVD
!
!
[Lee+99]](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-35-320.jpg)
![NMF = PLSI [Dinga+08]
!
NMF
PLSI
!
NMF
PLSI](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-36-320.jpg)
![Non-negative Tensor Factorization (NTF)
[Cruys10]
3
!
!
2
3](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-37-320.jpg)
![Neural Network Language Model (NNLM) [Bengio
+03]
!
!
N
NN
N-1](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-40-320.jpg)
![Recurrent Neural Network Language Model
(RNNLM) [Mikolov+10]
!
t-1
t
!
NNLM
N
!
!
http://rnnlm.org](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-41-320.jpg)
![RNNLM
!
[Mikolov+13a]](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-42-320.jpg)
![Skip-gram
(word2vec) [Mikolov+13b]
!
!
CBOW
!
Analogical reasoning
!
Parser](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-44-320.jpg)
![Skip-gram
[Mikolov+13b]
: w1, w2, …, wT
!
wi
c
vw
w
5](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-45-320.jpg)
![[Mikolov+13c]
!](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-47-320.jpg)
![[Kim+13]
!
“good”
”best”
”better”](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-49-320.jpg)
![[Mikolov+13d]
!
!](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-50-320.jpg)
![1
!
!
!
!
!
[Bird+10] Steven Bird, Ewan Klein, Edward Loper,
,
,
.
.
, 2010.
[
+96]
.
.
, 1996.
[Evert10] Stefan Evert.
Distributional Semantic Models. NAACL 2010 Tutorial.
[
13]
.
.
, 2013.
[Deerwester+90] Scott Deerwester, Susan T. Dumais, George W.
Furnas, Thomas K. Landauer, Richard Harshman.
Indexing by Latent Semantic Analysis. JASIS, 1990.](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-55-320.jpg)
![2
!
!
!
!
!
[Hofmann99] Thomas Hofmann.
Probabilistic Latent Semantic Indexing. SIGIR, 1999.
[Blei+03] David M. Blei, Andrew Y. Ng, Michael I. Jordan.
Latent Dirichlet Allocation. JMLR, 2003.
[Lee+99] Daniel D. Lee, H. Sebastian Seung.
Learning the parts of objects by non-negative matrix factorization.
Nature, vol 401, 1999.
[Ding+08] Chris Ding, Tao Li, Wei Peng.
On the equivalence between Non-negative Matrix Factorization and
Probabilistic Latent Semantic Indexing. Computational Statistics &
Data Analysis, 52(8), 2008.
[Cruys10] Tim Van de Cruys.
A Non-negative Tensor Factorization Model for Selectional Preference
Induction. Natural Language Engineering, 16(4), 2010.](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-56-320.jpg)
![3
!
!
!
!
NN 1
[Bengio+03] Yoshua Bengio, Réjean Ducharme, Pascal Vincent,
Christian Jauvin.
A Neural Probabilistic Language Model. JMLR, 2003.
[Mikolov+10] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan
"Honza" Cernocky, Sanjeev Khudanpur.
Recurrent neural network based language model.
Interspeech, 2010.
[Mikolov+13a] Tomas Mikolov, Wen-tau Yih, Geoffrey Zweig.
Linguistic Regularities in Continuous Space Word
Representations. HLT-NAACL, 2013.
[Mikolov+13b] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey
Dean.
Efficient Estimation of Word Representations in Vector Space.
CoRR, 2013.](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-57-320.jpg)
![4
!
!
!
NN 2
[Mikolov+13c] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory
S. Corrado, Jeffrey Dean.
Distributed Representations of Words and Phrases and their
Compositionality. NIPS, 2013.
[Kim+13] Joo-Kyung Kim, Marie-Catherine de Marneffe.
Deriving adjectival scales from continuous space word
representations. EMNLP 2013.
,
[Mikolov+13d] Tomas Mikolov, Quoc V. Le, Ilya Sutskever.
Exploiting Similarities among Languages for Machine
Translation. CoRR, 2013.](https://image.slidesharecdn.com/20140206statisticalsemantics-140205232927-phpapp02/85/Statistical-Semantic-word2vec-58-320.jpg)
Statistical Semantic入門 ~分布仮説からword2vecまで~
- 1. 2014/02/06 PFI Statistical Semantic ~ word2vec Preferred Infrastructure (@unnonouno) ~
- 4. Semantics
- 9. Statistical Semantics Distributional Semantics ! ! !
- 10. [Evert10] NAACL2010 Stefan Evert Semantic Models Distributional
- 11. ??? [Evert10]
- 13. dog [Evert10]
- 14. (Distributional Hypothesis) The Distributional Hypothesis is that words that occur in the same contexts tend to have similar meanings (Harris, 1954). (ACL wiki ) ! !
- 15. (Statistical Semantics) Statistical Semantics is the study of "how the statistical patterns of human word usage can be used to figure out what people mean, at least to a level sufficient for information access” (ACL wiki ) ! !
- 17. [ 13] ! ! !
- 18. ! ! ! ! ! ! ! !
- 21. : Latent Semantic Indexing (LSI), Latent Semantic Analysis (LSA) [Deerwester+90] ! ! ! !
- 23. LSI ! ! ! ! SVD
- 28. ! ! !
- 29. Probabilistic Latent Semantic Indexing (PLSI) [Hofmann99] ! LSI ! ! ! ex: LSI
- 31. Latent Dirichlet Allocation (LDA) [Blei03] PLSI ! PLSI LDA !
- 32. LDA ! NLP ! ! 1
- 35. Non-negative Matrix Factorization (NMF) [Lee +99] ! SVD ! ! [Lee+99]
- 36. NMF = PLSI [Dinga+08] ! NMF PLSI ! NMF PLSI
- 37. Non-negative Tensor Factorization (NTF) [Cruys10] 3 ! ! 2 3
- 38. ! ! SVD
- 40. Neural Network Language Model (NNLM) [Bengio +03] ! ! N NN N-1
- 41. Recurrent Neural Network Language Model (RNNLM) [Mikolov+10] ! t-1 t ! NNLM N ! ! http://rnnlm.org
- 45. Skip-gram [Mikolov+13b] : w1, w2, …, wT ! wi c vw w 5
- 46. !
- 47. [Mikolov+13c] !
- 53. ! NN ! ! ! ! ! !
- 55. 1 ! ! ! ! ! [Bird+10] Steven Bird, Ewan Klein, Edward Loper, , , . . , 2010. [ +96] . . , 1996. [Evert10] Stefan Evert. Distributional Semantic Models. NAACL 2010 Tutorial. [ 13] . . , 2013. [Deerwester+90] Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, Richard Harshman. Indexing by Latent Semantic Analysis. JASIS, 1990.
- 56. 2 ! ! ! ! ! [Hofmann99] Thomas Hofmann. Probabilistic Latent Semantic Indexing. SIGIR, 1999. [Blei+03] David M. Blei, Andrew Y. Ng, Michael I. Jordan. Latent Dirichlet Allocation. JMLR, 2003. [Lee+99] Daniel D. Lee, H. Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. Nature, vol 401, 1999. [Ding+08] Chris Ding, Tao Li, Wei Peng. On the equivalence between Non-negative Matrix Factorization and Probabilistic Latent Semantic Indexing. Computational Statistics & Data Analysis, 52(8), 2008. [Cruys10] Tim Van de Cruys. A Non-negative Tensor Factorization Model for Selectional Preference Induction. Natural Language Engineering, 16(4), 2010.
- 57. 3 ! ! ! ! NN 1 [Bengio+03] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin. A Neural Probabilistic Language Model. JMLR, 2003. [Mikolov+10] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan "Honza" Cernocky, Sanjeev Khudanpur. Recurrent neural network based language model. Interspeech, 2010. [Mikolov+13a] Tomas Mikolov, Wen-tau Yih, Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. HLT-NAACL, 2013. [Mikolov+13b] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. CoRR, 2013.
- 58. 4 ! ! ! NN 2 [Mikolov+13c] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. NIPS, 2013. [Kim+13] Joo-Kyung Kim, Marie-Catherine de Marneffe. Deriving adjectival scales from continuous space word representations. EMNLP 2013. , [Mikolov+13d] Tomas Mikolov, Quoc V. Le, Ilya Sutskever. Exploiting Similarities among Languages for Machine Translation. CoRR, 2013.