Downloaded 110 times
More Related Content
Similar to A Panorama of Natural Language Processing
A Panorama of Natural Language Processing
- 1. A Panorama of NaturalLanguage Processing Ted Xiao
- 2. Overview • Background • Grammars •Word Representation • Modern NLP • Future Directions • NLP in Industry • Demos
- 3. What is NaturalLanguage Processing? NLP! Artificial Languages: Java, C++, Binary…
- 4. What is NaturalLanguage Processing? NLP! Artificial Languages: Java, C++, Binary… Natural Language: Language spoken by people.
- 5. What is NaturalLanguage Processing? NLP! Artificial Languages: Java, C++, Binary… Natural Language: Language spoken by people. Motivation: Sophisticated linguistic analysis for human-like sophistication for a range of tasks or applications.
- 6. What is NaturalLanguage Processing? NLP! Goal: have computers understand natural language in order to perform useful tasks Artificial Languages: Java, C++, Binary… Natural Language: Language spoken by people. Motivation: Sophisticated linguistic analysis for human-like sophistication for a range of tasks or applications.
- 7. Task Types â—Ź Syntax â—‹Parsing â—‹ Stemming â—‹ Part of speech tagging â—Ź Discourse â—‹ Parsing â—‹ Stemming â—‹ Part of speech tagging
- 8. Task Types â—Ź Syntax â—‹Parsing â—‹ Stemming â—‹ Part of speech tagging â—Ź Semantics â—‹ Machine Translation â—‹ Natural Language Understanding, Generation â—‹ OCR â—‹ QA, Sentiment Analysis â—‹ Coreference â—Ź Discourse â—‹ Parsing â—‹ Stemming â—‹ Part of speech tagging â—Ź Speech â—‹ Speech Recognition â—‹ Text-to-Speech â—‹ Speech-to-Text
- 9.
- 10.
- 11.
- 12. What Makes NLPDifficult? • We don’t understand language ourselves • Language encodes meaning • Language is learned intuitively - easy for children, hard for computers or
- 13. What Makes NLPDifficult? • We don’t understand language ourselves • Language encodes meaning • Language is learned intuitively - easy for children, hard for computers • Ambiguity • Language is symbolic • Subtleties: sarcasm, wordplay, idioms... or
- 14. NLP vs. PLP ProgrammingLanguage Processing is easier than Natural Language Processing
- 15. ● The Pope’sbaby steps on gays ● Scientists study whales from space ● Juvenile court to try shooting defendant ● Boy paralyzed after tumor fights back to gain black belt Examples of ambiguity: news headlines
- 16.
- 17. An NLP Disaster: MicrosoftTay (March 2016)
- 18.
- 19. Overview • Background • Linguistics •Word Representation • Modern NLP • Future Directions • NLP in Industry • Demos
- 20. Grammars â—Ź Grammars arethe formal description of the structure of a language â—Ź Skeleton of any language
- 21.
- 22.
- 23. Overview • Background • Grammars •Word Representation • Modern NLP • Future Directions • NLP in Industry • Demos
- 24. Digitalizing Natural Language •Need to have some measure of similarity and differences between words • Vectors can do this!
- 25. Digitalizing Natural Language •Need to have some measure of similarity and differences between words • Vectors can do this! • We can use vector operations to gauge similarity between words
- 26. Word Vectors • 13million tokens in the English language • Many words are similar (cat and feline, man and woman, etc…) • A nice idea: encode word tokens into a vector that is a point in some word space with dimension << 13 million
- 27. One-Hot Vectors • Expresseach word as an |V| dimensional vector with one 1 and the rest 0s, where |V| is the size of our vocabulary • One-hot vectors for a dictionary would look like:
- 28. What’s Wrong? • Onehot vectors are independent (orthogonal) • But some words are similar! • A nicer idea: reduce the size of the space from |V| to a smaller- dimensional subspace that encodes relationships between words
- 29. Quick Aside: SingularValue Decomposition (SVD) (mxm) matrix of left singular vectors (nxn) matrix of right singular vectors (mxn) matrix with the singular values of X on its diagonals (mxn) Take away point: The SVD of X is the best rank-k approximation X = USVT
- 30. Illustration of theSVD as a Rank-k Approximation
- 31. SVD-Based Methods: Window-basedCo- occurrence Matrix • Only count the number of times a word appears inside a window of a particular size around the word of interest • Consider the following three documents: • I enjoy flying • I like NLP • I like deep learning window size = 1
- 32. Applying the SVDto the co- occurrence matrix • X = USVT • Truncate S at some index k based on the amount of variance captured: • Take the sub-matrix Ui:V,1:k to be our word embeddings • Now have a k-dimensional representation of every word in our vocabulary!
- 33. Downfalls of SVD-basedMethods • Co-occurrence matrix is high dimensional and sparse • SVDs are computationally expensive (quadratic cost) • Dimensions of the co-occurrence matrix are constantly changing as new words are added
- 34. Downfalls of SVD-basedMethods • Co-occurrence matrix is high dimensional and sparse • SVDs are computationally expensive (quadratic cost) • Dimensions of the co-occurrence matrix are constantly changing as new words are added • Solution: iteration-based methods!
- 35. Iteration-based Methods • WordVectors and Word Embeddings: Used to Find similarity Compute and store representative information about a huge dataset • Iteration-based Methods: create a model that learns one iteration at a time that will eventually be able to encode the probability of a word given its context • These include basic language models as well as the more advanced word2vec
- 36. Basic Language Models •Bag of Words • Just count the frequencies of words. • Issues: High dimension, and order and relations are lost
- 37. Basic Language Models •Bag of Words • Just count the frequencies of words. • Issues: High dimension, and order and relations are lost • Term Frequency-inverse Document Frequency • AKA TF-IDF • How important a word is in a document • Used in search engines!
- 38. Basic Language Models •Question: Are there ways we can maintain information about word order and meaning? • Bag of Words • Just count the frequencies of words. • Issues: High dimension, and order and relations are lost • Term Frequency-inverse Document Frequency • AKA TF-IDF • How important a word is in a document • Used in search engines!
- 39. Language Models • Goal:assign a probability to a sequence of tokens • Consider the two sentences: • “The dog wagged his tail” • “Puffer fish bank ladder” • Which should have a higher probability? • If we assume that word occurrences are independent, the probability of any given sequence of words is: (Unigram model)
- 40. What if WordOccurrences Are Not Independent? • Assume the probability of a sequence depends on the pairwise probability of a word in the sequence and a word next to it (bigrams) • A bigram model is of the form: • The general N-gram model is given by:
- 41. Approaches So Far â—ŹSimple models trained on huge amounts of data outperform complex models trained on small amounts of data â—Ź Unigrams: â—Ź Bigrams: â—Ź N-grams:
- 42. Continuous Bag ofWords (CBOW) • Consider part of the sequence of words as context, and try to predict the center words • Sentence: “The dog wagged his tail” • Context: {“The”, “dog”, “his”, “tail} • Center words: “wagged” • Our known parameters are the sentence in question represented by one-hot vectors • Let x(c) denote the context words • Lets y(c) denote the target word (output)
- 43. Continuous Bag ofWords
- 44. Skip-grams • Now, considerthe center word as context and try to predict surrounding words • Sentence: “The dog wagged his tail” • Context: “wagged” • Surrounding words: {“The”, “dog”, “his”, “tail”} • Nearly identical set-up to CBOW, except we switch our x and y • Input: one-hot vector (context word) • Output: vectors describing the surrounding words
- 45.
- 46. Recap ● We firsttried condensing language into word vectors ○ We want to keep meaning in a lower dimension ○ One-hot Vectors, SVD… These are expensive!
- 47. Recap ● We firsttried condensing language into word vectors ○ We want to keep meaning in a lower dimension ○ One-hot Vectors, SVD… These are expensive! ● We then tried iteration based methods ○ Language Models: Bag of Words, TF IDF… no context!
- 48. Recap ● We firsttried condensing language into word vectors ○ We want to keep meaning in a lower dimension ○ One-hot Vectors, SVD… These are expensive! ● We then tried iteration based methods ○ Language Models: Bag of Words, TF IDF… no context! ● We add in context with N-gram models
- 49. Recap ● We firsttried condensing language into word vectors ○ We want to keep meaning in a lower dimension ○ One-hot Vectors, SVD… These are expensive! ● We then tried iteration based methods ○ Language Models: Bag of Words, TF IDF… no context! ● We add in context with N-gram models ● We extended these with CBOW and Skip-grams ○ CBOW: Predict center word ○ Skip-grams: Predict surrounding words
- 50. • So far,we have an expression for the chance that a sequence of words appears as products of conditional probabilities • Now, we’d like models that can learn the probabilities of word- sequences • Solution: Word2vec (Mikolov et al, 2013) Learning word-sequence probabilities
- 51. Word2vec • A neuralnetwork implementation that learns distributed representations for words • 2 algorithms • Continuous bag of words • Skip-grams • 2 training methods • Negative sampling • Hierarchical softmax • Best part: DOES NOT NEED LABELED DATA!
- 52. Word2vec • Many ofthose steps are complicated... • Luckily, someone made software that does this for us • Gensim is a python package that can do all of this complicating word2vec stuff in a few lines of code • Results: Training high dimensional word vectors on a large amount of data captures “Subtle semantic relationships between words”
- 53. Reflecting on word2vec •Words with similar meanings occur in clusters • Clusters are spaced such that some word relationships (such as analogies) can be reproduced with vector math • Famous example (with highly trained word vectors) • “king” - “man” + “woman” = “queen” • Useful feature: word2vec does not require labeled data • Most data in the world is unlabeled! • Word embeddings are very useful for prediction and translation tasks, as well as sentiment analysis
- 54. Overview • Background • Grammars •Word Representation • Modern NLP • Future Directions • NLP in Industry • Demos
- 55. Modern NLP • Advancesin NLP were largely driven by • A vast increase in computing power • A better understanding of human language • Development of successful ML algorithms • Big data • Much of current work involves: • Machine translation • Spoken dialogue and conversational agents • Machine reading • Mining social media • Analysis and generation of speaker state
- 56. Forms of NLPData User data Corpora Dictionaries Ontologies and databases
- 57. NLP Data Sources •Wikimedia • APIs: Twitter, Wordnik, … • Common crawl • Wordnet • Linguistic data consortium (www.ldc.upenn.edu) • University sites and the academic community • Stanford, Oxford, CMU • Create your own! • Web-scrape, crowd-source, linguists
- 58. Deep Learning vsNon-deep Learning Methods • Bag of words may outperform deep learning models in modest sized datasets • Word2vec sees a drastic improvement with a LOT of text • In literature, distributed word vector techniques outperform bag of words models Deep learning tries to capture the recursive nature of natural language
- 59. Deep Learning forNLP • Deep learning attempts to learn multiple levels of representation of increasing complexity and abstraction • Want computers to be able to understand the recursive nature of human language • Recursive/recurrent neural networks! • DL models can be fast ways to solve NLP tasks
- 60. Recurrent Neural Network â—ŹRecurrent Neural Network â—‹ Connections between units form are directed cycles â—‹ Internal state of the network allows it to exhibit dynamic temporal behavior â—‹ Success in speech recognition, natural language, translation, etc. â—Ź Long Short-term Memory: LSTM
- 61.
- 62. seq2seq â—Ź Applied RNNsto Sequences â—‹ Generate a response based on meaningful input â—‹ For example, translate from English to French â—Ź Two RNNs: an encoder that processes the input and a decoder that generates the output.
- 63. Recursive Deep Learning •Compositional vector grammars (parsing) • Recursive autoencoders (paraphrase detection) • Matrix-vector RNNs (relation classification) • Recursive neural tensor networks (sentiment analysis)
- 64. What’s at UCBerkeley? • Berkeley NLP Research - Dan Klein • Computer Vision - Alexei Efros, Jitendra Malik CV + NLP: Visual Question Answer
- 65. Overview • Background • Grammars •Word Representation • Modern NLP • Future Directions • NLP in Industry • Demos
- 66. What the (near)future holds Bots! Think Siri, but actually functional instead of a toy
- 67. What the (near)future holds Supporting invisible UI! The concept of invisible or zero user interaction between user and machine
- 68. What the (near)future holds Smarter search! The same capabilities that allow a chatbox to understand a customer’s request can enable “search like you talk” functionality
- 69. What the (near)future holds Intelligence from unstructured information! Analysis that accurately understands the subtleties of natural language (choice of words, tone, etc) can provide useful knowledge and insight of information
- 70. Overview • Background • Grammars •Word Representation • Modern NLP • Future Directions • NLP in Industry • Demos
- 71.
- 73. NLP Architectures • LayeredModel • Preprocessing • Low-level analysis • Semantic Analysis • Conversion to end products • Input/Output as API Structure
- 74. NLP at Scale •Systems come before algorithms • Objective functions are messy • Everything is changing • Understanding-optimization trade-off
- 75. NLP at Scale •Systems come before algorithms • Objective functions are messy • Everything is changing • Understanding-optimization trade-off
- 76. Developing an NLPsystem 1. Exploration a. Translate real-world requirements into a measurable goal b. Find an appropriate level and representation c. Find data for experiments
- 77. Developing an NLPsystem 1. Exploration a. Translate real-world requirements into a measurable goal b. Find an appropriate level and representation c. Find data for experiments 2. Development a. Find and utilize existing tools and frameworks b. Set up and perform a series of experiments
- 78. Developing an NLPsystem 1. Exploration a. Translate real-world requirements into a measurable goal b. Find an appropriate level and representation c. Find data for experiments 2. Development a. Find and utilize existing tools and frameworks b. Set up and perform a series of experiments 3. Production a. CPU/GPU intensive b. Most NLP frameworks are not production-ready c. Pre- and post- processing is invaluable d. Collect user feedback
- 79. I Have theModel… Now What? 1. Specify Performance Requirements 2. Separate Prediction Algorithm From Model Coefficients a. Select or Implement The Prediction Algorithm b. Serialize Your Model Coefficients 3. Develop Automated Tests For Your Model 4. Develop Back-Testing and Now-Testing Infrastructure 5. Challenge Then Trial Model Updates
- 80. Tips for NLP •Proper preprocessing is VERY important • Know your domain! • Validate your models! • Human judges • Cross-validation
- 81. Overview • Background • Grammars •Word Representation • Modern NLP • Future Directions • NLP in Industry • Demo
- 82. Programming Language Identification Exploringcode on GitHub Goal is to figure out what language a file uses Potential Methods? Filename, keywords, comments Whitespace, syntax Must be scalable to handle large number of constantly updating repos
- 83. Existing Model Linguist: Heuristics+ Naive Bayes Heuristics can be accurate but require updating and fine-tuning Naive Bayes depends on word frequencies - predictions are linear to vocab size Hard-coded rules do most of the work, leaving the Naive Bayes as a last resort Heavily dependent on file extension classification Selective Classification With file extensions, only 87% of files are classified
- 86. Thank you! [email protected] Special thanksto Jordan Prosky
- 87.
- 88. Appendix • Language ismeant to convey meaning, which we have a natural way of encoding • Children learn this very fast! • Hard for computers to learn… • Language is a symbolic signaling system • Example: pen: or ? • Other subtleties: sarcasm, expressive signaling, …
- 89. What makes NLPdifficult? • Language is meant to convey meaning, which we have a natural way of encoding • Children learn this very fast! • Hard for computers to learn… • Language is a symbolic signaling system • Example: pen: or ? • Other subtleties: sarcasm, expressive signaling, …
- 90. Basics of NLPdata preprocessing • Domain specific! • Tokenization • Example: “This is a test that isn’t so simple” • Tokens: “This”, “is”, “a”, “test”, “that”, “is”, “n’t”, “so”, “simple” • Regular expressions • Stemming • Lower-casing • Removing/adding punctuation • Other…
- 91. SVD-Based Methods • Loopover a massive dataset to accumulate word co-occurrence counts in some matrix X • Perform the SVD on X to get U, S, and V • Use the rows of U as the word embeddings for all words in your dictionary X = ?
- 92. SVD-Based Methods: Word- DocumentMatrix • Assumption: related words often appear in the same document • Loop over many documents and every time word i appears in document j, add one to entry Xij • Very high dimensional - let’s try something better
- 93. We are notquite done… • Need to find suitable U and V matrices! • Two algorithms help us get what we want: • Hierarchical softmax • Negative sampling • These are complicated! • Luckily, someone made software that does this for us • Gensim is a python package that can do all of this complicating word2vec stuff in a few lines of code