Scribd
Upload
7 views20 pages

chap2part2

The document discusses token normalization in information retrieval, focusing on creating equivalence classes for terms to facilitate matching despite superficial differences. It covers methods such as asymmetric expansion, handling accents in different languages, and techniques like stemming and lemmatization to reduce words to their base forms. Additionally, it introduces the concept of skip pointers in postings for faster query processing and the trade-offs involved in their placement.

Uploaded by

oh74743
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
7 views20 pages

chap2part2

The document discusses token normalization in information retrieval, focusing on creating equivalence classes for terms to facilitate matching despite superficial differences. It covers methods such as asymmetric expansion, handling accents in different languages, and techniques like stemming and lemmatization to reduce words to their base forms. Additionally, it introduces the concept of skip pointers in postings for faster query processing and the trade-offs involved in their placement.

Uploaded by

oh74743
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 20

Normalization to terms

 Token normalization is the process of canonicalizing tokens so


that matches occur despite superficial differences in the character
sequences of the tokens.
 The most standard way to normalize is to create equivalence
classes , which are normally named after one member of the set.

 For Example: anti- discriminatory and antidiscriminatory are both


mapped onto the term antidiscriminatory, in both the document
text and queries, then searches for one term will retrieve
documents that contain either.

1
Creating Equivalence Classes
We most commonly implicitly define equivalence classes of terms by, e.g.,
deleting periods to form a term
U.S.A., USA  USA
deleting hyphens to form a term
anti-discriminatory, antidiscriminatory map to antidiscriminatory

• Maintaining relations between unnormalized tokens


– Indexing unnormalized tokens and maintain a query expansion list of
multiple vocabulary entries: when a query asks for “car”, search both
“car” and“automobile”
– Expansion during index construction: index a document containing “car”
under “car” and “automobile”
– Expansion of query terms can be asymmetric

– A more space costly but more flexible method

J. Pei: Information Retrieval and Web Search -- Tokenization 13


Sec. 2.2.3

Normalization to terms

• An alternative to equivalence classing is to do


asymmetric expansion
• An example of where this may be useful
– Enter: window Search: window, windows
– Enter: windows Search: Windows, windows, window
– Enter: Windows Search: Windows
Sec. 2.2.3

Normalization: other languages


• Accents: e.g., French résumé vs. resume.
• German: Tuebingen vs. Tübingen
– Should be equivalent
• Most important criterion:
– How are your users like to write their queries for
these words?
• Even in languages that standardly have
accents, users often may not type them
– Often best to normalize to a de-accented term
• Tuebingen, Tübingen, Tubingen  Tubingen
Sec. 2.2.3

Capitalization/case-folding
Reduce all letters to lower case
exception: upper case in mid-sentence?
e.g., General Motors
Fed vs. fed(governmental organization)
SAIL vs. sail

– Use some heuristics to make some tokens lowercase,


e.g., covert the first word in a sentence, all words in a
title

– Truecasing: use a machine learning sequence model to


make the decision

– Often best to lower case everything, since users


will use lowercase regardless of ‘correct’
Introduction to
Information Retrieval
Stemming and Lemmatization
Stemming and Lemmatization
• How can we know “organize”, “organizes”, and
“organizing” should map to the same word?
• Stemming and lemmatization: reduce inflectional
forms and sometimes derivationally related forms
of a word to a common base form
– am, are, is  be
– car, cars, car’s, cars’  car

– “the boy’s cars are different colors”  “the boy car be


different color”

J. Pei: Information Retrieval and Web Search -- Tokenization 16


8
Stemming
• Algorithmic: a crude heuristic process that chops
off the ends of words in the hope of being correct
most of the time
– Often remove derivational affixes

• Porter’s algorithm
 Conventions + 5 phases of reductions
 phases applied sequentially
 each phase consists of a set of rules

J. Pei: Information Retrieval and Web Search -- Tokenization 17


Sec. 2.2.4

Porter’s algorithm

Question:
circus
canaries
boss
ponies

measure of a word:
which check the number of syllables to see if the word is long
enough to regard the matching portion of a rule as a suffix rather
than as part of the stem of word

Use “(m>1) EMENT  ” to map replacement to replac


Cement isn’t mapped
Lemmatization
• Dictionary-based stemming
• Use a vocabulary and morphological analysis of
words to remove inflectional endings only and
return the base or dictionary form of a word
(lemma)
• “saw”  “see” or “saw” depending on whether
the token is used as a verb or a noun
• Can bring very modest benefit for retrieval in
English – improving recall but may hurt accuracy
e.g: query entered “ operating and system” not good matching
operate and system

J. Pei: Information Retrieval and Web Search -- Tokenization 19


12
Introduction to
Information Retrieval
Faster postings merges:
Skip pointers/Skip lists
Sec. 2.3

Recall basic merge


• Walk through the two postings
simultaneously, in time linear in the total
number of postings entries
2 4 8 41 48 64 128 Brutus
2 8
1 2 3 8 11 17 21 31 Caesar

If the list lengths are m and n, the merge takes O(m+n)


operations.

Can we do better?
Yes (if the index isn’t changing too fast).
Sec. 2.3

Augment postings with skip


pointers (at indexing time)
41 128
2 4 8 41 48 64 128

11 31
1 2 3 8 11 17 21 31

• Why?
• To skip postings that will not figure in the
search results.
• How?
• Where do we place skip pointers?
Sec. 2.3

Query processing with skip


pointers
41 128
2 4 8 41 48 64 128

11 31
1 2 3 8 11 17 21 31

Suppose we’ve stepped through the lists until we process 8 on each list. We match it
and advance.

We then have 41 and 11 on the lower. 11 is smaller.

But the skip successor of 11 on the lower list is 31, so


we can skip ahead past the intervening postings.
Consider a postings intersection between this postings list, with skip
pointers:

24 75 92 115
and the following intermediate result postings list (which hence has no
skip pointers):

3 5 89 95 97 99 100 101

a. How often is a skip pointer followed?

b. How many postings comparisons will be made by this algorithm while


intersecting the two lists?

c. How many postings comparisons would be made if the postings lists are
intersected without the use of skip pointers?

17
Sec. 2.3

Where do we place skips?


• Tradeoff:
– More skips  shorter skip spans  more likely to
skip. But lots of comparisons to skip pointers.
– Fewer skips  few pointer comparison, but then
long skip spans  few successful skips.
Sec. 2.3

Placing skips
•

Transfer time from desk will grow by


adding skip pointers
20

You might also like