Machine Learning Natural Language

1
 NLP System : IBM Watson

Question Answering System

Quiz show Jeopardy!

• “The first person mentioned by name in 'The man in the Iron

mask' is this hero of a previous book by the same author”

2
 Natural Language Processing
• NLP focuses on developing systems that allow
computers to perform useful tasks involving human
language
– Also called Computational Linguistics
• NLP applications
– Information Retrieval
– Question Answering
– Machine Translation
– Information Extraction

3
 NLP application : Information Retrieval
• Stemming
• Spell checking
• Query expansion
• Word sense
disambiguation

6
 NLP application : Question Answering

• Determine type of question and answer

• Parse the question and identify relations
POS tagging, Parsing, named entity recognition

7
 NLP application : Machine Translation

• Sentence alignment
• POS tagging
• Parsing
• Sentence generation grammars
• Named Entity Recognition (“New Delhi”)

8
NLP application : Information Extraction

• Identifying/Extracting specific kinds of information

• Named entities (NEs): person, location, price, product
– Mohandas Karamchand Gandhi was born in Porbandar, Gujarath
• Coreference resolution: linking pronouns/abbreviations to entities
– “Indian Institute of Science” <> “IISc.”
• Relations: <DOB>, <spouse>, <attribute>

27
 NLP application : Categorization
• Topical : politics, sports, business
• Sentiment: positive, negative, neutral
POS tagging to obtain adjectives

28
 NLP : Tasks

• Segmentation : words, sentences

● Morphology : plural “boy” “boys” , “agree” ---> “agreement”
Stemming "fishing", "fished", "fish", "fisher" ---> "fish"
• Syntactic Analysis : structural relationships between words
– Part of Speech (POS) Tagging
Machine[N] learning[V] natural[Adj] language[N]
– Parsing

Machine[N] learning[V] natural[Adj] language[N]

 NLP : Tasks
• Semantics
– Word Sense Disambiguation : “I went to bank”
– Semantic role labelling :
“Mary[Agent] sold the book[goods] to John[Recepient]”

• Pragmatics : how language is used to accomplish goals

– I’m sorry Dave, I’m afraid I can’t do that [Polite]
– I can't do that [Rude]

• Discourse
Coreference Resolution : linking pronouns/abbreviations to entities
“I saw Scott yesterday. He was fishing by the lake.”
“Indian Institute of Technology Hyderabad is a public institution
located in Hyderbad. IITH was established in 2007.”

Named Entity recognition (NER) : person, location, price, product

Mohandas Karamchand Gandhi was born in Porbandar, Gujarath
 NLP is hard

● Natural language is ambigious

● Sentence Segmentation : “I went out with Mr. Smith.”
• Syntactic
“Flies[Noun/Verb] like flower[Noun/Verb]”

“I saw the man with the telescope” vs

“I saw the man with the telescope”

• Semantic
“I put the plant in the window” vs “Ford put the plant in Mexico”
• Ambiguity is Explosive
“I saw the man on the hill with the telescope.”: 4 parses
 Machine Learning Natural Language
● “Rules” in language have numerous exceptions and irregularities
● Manual knowledge engineering, is difficult, time-consuming, and error
prone.
●
Use machine learning methods to automatically acquire the required
knowledge from appropriately annotated text corpora.
●
Annotating corpora is easier and requires less expertise than manual
knowledge engineering.
 Machine Learning POS Tagging
•
Lowest level of syntactic analysis
• Useful for Parsing and word sense disambiguation
• Ambiguity in POS tagging
Flies[Noun] like[Verb] flower[Noun]
Time flies[Verb] like[Prep] an arrow.

Learning : Train models on human annotated corpora like the Penn Treebank.
 POS Tagging
Classification

Classify each word independently but use as input features,

information about the surrounding words.

Time flies like an arrow.

classifier

NN

14
 POS Tagging
Classification
NN
Time flies like an arrow.

classifier

VBZ

15
 POS Tagging
● Classification
NN VBZ
Time flies like an arrow.

classifier

VBP

16
 POS Tagging
● Classification
NN VBZ VBP
Time flies like an arrow.

classifier

DT

17
 POS Tagging
Classification
NN VBZ VBP DT
Time flies like an arrow.

classifier

NN

18
 POS Tagging
Classification
NN VBZ VBP DT NN
Time flies like an arrow.

Sequence Labeling
Tags of words are dependent on the tags of other words in
the sentence, particularly their neighbors

Time flies like an arrow.

classifier

NN
19
 POS Tagging
Classification
NN VBZ VBP DT NN
Time flies like an arrow.

Sequence Labeling
NN
Time flies like an arrow.

classifier

VBZ

20
 POS Tagging
Classification
NN VBZ VBP DT NN
Time flies like an arrow.

Sequence Labeling
NN VBZ
Time flies like an arrow.

classifier

IN

21
 POS Tagging
Classification
NN VBZ VBP DT NN
Time flies like an arrow.

Sequence Labeling
NN VBZ IN
Time flies like an arrow.

classifier

DT

22
 POS Tagging
Classification
NN VBZ VBP DT NN
Time flies like an arrow.

Sequence Labeling
NN VBZ IN DT
Time flies like an arrow.

classifier

NN

23
 Sequence Labeling
Classification
NN VBZ VBP DT NN
Time flies like an arrow.
Sequence Labeling

NN VBZ IN DT NN
Time flies like an arrow.

POS Tagging is best modeled as a sequence learning problem than as

a classification problem
- Information Extraction, Named Entity recognition

Statistical models: Hidden Markov Model (HMM), Maximum Entropy Markov

Model (MEMM), Conditional Random Field (CRF)

24
 Parsing

•
Ambiguity
“I saw the man with the telescope” vs
“I saw the man with the telescope”
Probabilistic Context Free Grammars (PCFG)

• Structured Prediction

Machine learning natural language

Strings Trees

Statistical models: Conditional Random Field, Structured perceptrons, Structured support

vector machines
25
 Machine learning for NLP

• Transfer Learning, domain adaptation

– Adapting a model learned on a resource rich language to
resource scarce language
• Deep learning
– Unsupervised learning of useful features

● Conferences : Association of Computational Linguistics(ACL),

Computational Linguistics (COLING), Empirical Methods in NLP (EMNLP)

• Software tools
Stanford CoreNLP, openNLP, NLTK, Lingpipe

26
 References
Daniel Jurafsky and James H. Martin (2008). Speech and Language Processing

Christopher D. Manning and Hinrich Schütze (1999). Foundations of Statistical

Natural Language Processing.

Machine Learning Methods in Natural Language Processing

http://www.cs.columbia.edu/~mcollins/papers/tutorial_colt.pdf

Lafferty, J., McCallum, A., Pereira, F. (2001). Conditional random fields:

Probabilistic models for segmenting and labeling sequence data.

Ioannis Tsochantaridis, Thorsten Joachims, Thomas Hofmann and Yasemin Altun

(2005), Large Margin Methods for Structured and Interdependent Output Variables

Deep learning for NLP,

http://www.socher.org/index.php/DeepLearningTutorial/DeepLearningTutorial
Thank you