Scribd
Upload
1K views3 pages

Case Study of Lexical Analyzer PDF

Lexical analysis is the first phase of compiling source code. It breaks the source code into tokens by removing whitespace and comments. The lexical analyzer checks for valid tokens and generates errors if invalid tokens are found. It identifies tokens and expands macros while reading input characters from the source code. Common lexical errors include misspelled identifiers or keywords. Error recovery techniques include removing or ignoring characters. Lexical analysis improves compiler efficiency by eliminating unwanted tokens and allowing specialization through a separate lexical analyzer.

Uploaded by

AMIT DHANDE
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
1K views3 pages

Case Study of Lexical Analyzer PDF

Lexical analysis is the first phase of compiling source code. It breaks the source code into tokens by removing whitespace and comments. The lexical analyzer checks for valid tokens and generates errors if invalid tokens are found. It identifies tokens and expands macros while reading input characters from the source code. Common lexical errors include misspelled identifiers or keywords. Error recovery techniques include removing or ignoring characters. Lexical analysis improves compiler efficiency by eliminating unwanted tokens and allowing specialization through a separate lexical analyzer.

Uploaded by

AMIT DHANDE
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

Case Study: Lexical Analyzer

Lexical analysis is the first phase of a compiler. It takes the modified source code from language
preprocessors that are written in the form of sentences. The lexical analyzer breaks these
syntaxes into a series of tokens, by removing any whitespace or comments in the source code.

If the lexical analyzer finds a token invalid, it generates an error. The lexical analyzer works
closely with the syntax analyzer. It reads character streams from the source code, checks for
legal tokens, and passes the data to the syntax analyzer when it demands.

Roles of the Lexical analyzer


Lexical analyzer performs below given tasks:

1. Helps to identify token into the symbol table


2. Removes white spaces and comments from the source program
3. Correlates error messages with the source program
4. Helps you to expands the macros if it is found in the source program
5. Read input characters from the source program

Lexical Errors
A character sequence which is not possible to scan into any valid token is a lexical error.
Important facts about the lexical error:

1. Lexical errors are not very common, but it should be managed by a scanner
2. Misspelling of identifiers, operators, keyword are considered as lexical errors
3. Generally, a lexical error is caused by the appearance of some illegal character, mostly at
the beginning of a token.
Error Recovery in Lexical Analyzer
Here, are a few most common error recovery techniques:

1. Removes one character from the remaining input


2. In the panic mode, the successive characters are always ignored until we reach a
well-formed token
3. By inserting the missing character into the remaining input
4. Replace a character with another character
5. Transpose two serial characters

Why separate Lexical and Parser?

1. The simplicity of design: It eases the process of lexical analysis and the syntax analysis
by eliminating unwanted tokens
2. To improve compiler efficiency: Helps you to improve compiler efficiency
3. Specialization: specialized techniques can be applied to improves the lexical analysis
process
4. Portability: only the scanner requires to communicate with the outside world
5. Higher portability: input-device-specific peculiarities restricted to the lexer

Advantages of Lexical analysis

1. Lexical analyzer method is used by programs like compilers which can use the parsed
data from a programmer's code to create a compiled binary executable code
2. It is used by web browsers to format and display a web page with the help of parsed data
from JavaScript, HTML, CSS
3. A separate lexical analyzer helps you to construct a specialized and potentially more
efficient processor for the task
Disadvantage of Lexical analysis

1. You need to spend significant time reading the source program and partitioning it in the
form of tokens
2. Some regular expressions are quite difficult to understand compared to PEG or EBNF
rules
3. More effort is needed to develop and debug the lexer and its token descriptions
4. Additional runtime overhead is required to generate the lexer tables and construct the
tokens

Diagram:

Conclusions: ​In ​computer science​, lexical analysis, lexing or tokenization is the process of
converting a sequence of characters such as in a computer program or web page into a sequence
of tokens, ​strings with an assigned and thus identified meaning. A program that performs lexical
analysis may be termed a lexer, tokenizer, or scanner, though scanner is also a term for the first
stage of a lexer.Thus we have successfully studied it.

You might also like