1/20/23, 11:54 PM Source Coding Theorem

Source Coding Theorem

The Code produced by a discrete memoryless source, has to be efficiently represented, which is an
important problem in communications. For this to happen, there are code words, which represent
these source codes.

For example, in telegraphy, we use Morse code, in which the alphabets are denoted by Marks and
Spaces. If the letter E is considered, which is mostly used, it is denoted by “.” Whereas the letter Q
which is rarely used, is denoted by “--.-”

Let us take a look at the block diagram.

Where Sk is the output of the discrete memoryless source and bk is the output of the source
encoder which is represented by 0s and 1s.

The encoded sequence is such that it is conveniently decoded at the receiver.

Let us assume that the source has an alphabet with k different symbols and that the kth symbol Sk
occurs with the probability Pk, where k = 0, 1…k-1.

Let the binary code word assigned to symbol Sk, by the encoder having length lk, measured in bits.

Hence, we define the average code word length L of the source encoder as

k−1
¯
¯¯¯
L = ∑ pk lk

k=0

L represents the average number of bits per source symbol

https://www.tutorialspoint.com/digital_communication/digital_communication_source_coding_theorem.htm 1/2
1/20/23, 11:54 PM Source Coding Theorem

¯
¯¯¯
If Lmin = minimum possible value of L

Then coding efficiency can be defined as

Lmin
η =
¯
¯¯¯
L

¯
¯¯¯
With L ≥ Lmin we will have η ≤ 1

However, the source encoder is considered efficient when η = 1

For this, the value Lmin has to be determined.

Let us refer to the definition, “Given a discrete memoryless source of entropy H (δ) , the average

¯
¯¯¯
code-word length L for any source encoding is bounded as L ≥ H (δ) ."

In simpler words, the code word

example : M orsecodef orthewordQU EU Eis−. −. . −. . . −. is always greater than or

equal to the source code QU EU Einexample . Which means, the symbols in the code word

are greater than or equal to the alphabets in the source code.

Hence with Lmin = H (δ) , the efficiency of the source encoder in terms of Entropy H (δ) may

be written as

H (δ)
η =
¯
¯¯¯
L

This source coding theorem is called as noiseless coding theorem as it establishes an error-free
encoding. It is also called as Shannon’s first theorem.

https://www.tutorialspoint.com/digital_communication/digital_communication_source_coding_theorem.htm 2/2
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
KTUStudents.in

For more study materials: WWW.KTUSTUDENTS.IN

Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
WEINER HOPF PROBLEM