Error-Control Coding

(Sklar: chapter 6, 7; Lin and Costello:

Error Control Conding: Fundamentals
and Applications, 1983, Chapters
3,4,6,10,11)

Terminal Connectivity

(a) Simplex: transmission in only one

direction.
(b) Half duplex: transmission can be in
either direction but not
simultaneously.
(c) Full Duplex: transmission can be in
both directions simultaneously.

76a 76b
Types of Error Control (a) Stop-and-wait ARQ, requires half-
duplex link
Error detection and retransmission (b) Continuous ARQ with pullback; it
In this mode the receiver detects errors requires full duplex.
in the received data but cannot correct (c) Continuous ARQ with selective
them. Instead it asks the transmitter to repeat; it requires full duplex.
retransmit erroneous part of the data
sequence. This method requires a return
link.

Forward error correction (FEC)

In this mode the receiver detects errors
and also correct them. This method does
not require a return link.

Automatic Repeat Request (ARQ)

This is the first type of error control. See
the figure below.
77a 77b
Error-control Codes • From the point of view of coding
theory, we are interested in how well
• Redundant bits are added to the the code improves the bit error
message bits (coding) in order for the probability or BER.
receiver to detect (ARQ) or detect and • Without coding the BER is computed
correct (FEC) the errors. by comparing the input data sequence
• There are two categories of codes: of the modulator with the output data
block codes and convolutional codes. sequence of the demodulator.
A recent developed class of codes is • &KDQQHO'HILQLWLRQ: with coding, the
the turbo codes that are based on the
modulator input is connected to the
convolutional codes.
encoder output and the decoder input is
connected to the demodulator output
Channel Models (See Sklar’s feature block diagram on
his book cover). Therefore we consider
• Before we discuss various codes, we all communication blocks between the
need to define models of channels that channel encoder and the channel
will be used in the coding theory. decoder as ONE block, called channel.
This “channel” actually includes
78a 78b
 modulator, transmitter, antennas (if Binary Symmetrical Channel (BSC)
any), the real channel (space, cable,
wire, storage medium, etc.), and • The figure “1.5” (a) below is the
demodulator (See the figure below). graphical representation of a BSC.
This “channel” model can have the
following three cases.

• 7UDQVLWLRQ3UREDELOLWLHV: The input

data bits (0,1) are sent through the
channel with transition probability of p
or 1-p. We denote the transition
probabilities as
79a 79b
 P(0 |1) = P(1| 0) = p Discrete Memoryless Channel (DMC)
P(1|1) = P(0 | 0) = 1 − p A better decoding is to use quantized
where P( j | i ) denotes the conditional demodulator output as the input to the
probability of detecting symbol j when decoder. Then the channel model is
symbol i is actually transmitted. described by the model in Fig. “1.5(b)”.
Apparently p is the BER of the The transition probabilities are
demodulator. For example, if the
modulation is BPSK, then P( j | i ), 0 ≥ i ≤ M − 1, 0 ≥ j ≤ Q − 1,
( )
p = Q 2 Eb / N 0 .
• +DUGGHFLVLRQGHFRGHU 
where M is the size of the input symbol
alphabet and Q is the size of the output
For BSC model, the demodulator symbol alphabet. The channel has no
output is firmly either 1 or 0, no other memory, meaning the each output
values are allowed, this type of
demodulator is called KDUGGHFLVLRQ
symbol only depends on one input
symbol. BSC is a special case of DMC
demodulator. The decoder accepts the when M=Q=2.
hard-decision data is called KDUG
GHFLVLRQGHFRGHU
80a 80b
Gaussian channel in the Gaussian channel model it is
• $OLPLWLQJFDVHRID'0& is the
normalized so that the signal part is just
the transmitted symbol. Thus the pdf of
Gaussian channel. In this case, the the Gaussian channel sufficient statistic z
output has an infinite number of levels, conditioned on the transmitted symbol uk
or analog values. In fact, this means is given by
that the demodulator make no decision 2  ( z − uk ) 2 
at all. For example, if modulation is p ( z | uk ) = exp  − 
BPSK (see Figure “4.8” on p. 4a), the σ 2π  2 σ 2

decoder directly takes as its input the
output of the integrator that is the • $SSUR[LPDWLRQ Gaussian channel can
“sufficient statistic” consisting of a only be approximated with digital
signal component and a Gaussian noise circuit by making the levels of
part. The “Decision Stage” is quantization very large (for example,
eliminated. 8-bit quantization that produces 256
• *DXVVLDQFKDQQHOFKDUDFWHUL]DWLRQ levels.)
The signal part in the aforementioned • 6RIWGHFLVLRQGHFRGHUWhen the
“sufficient statistic” is proportional to input of the modulator is binary, but
the transmitted symbol. For simplicity, the output of the demodulator is Q-ary
81a 81b
 (Q>2) or left unquantized, the smallest BER, which serves as a
demodulator is called VRIWGHFLVLRQ benchmark for other channels.
demodulator. The decoder accepts the
soft-decision data is called VRIW
Binary Field and Vector
GHFLVLRQGHFRGHUObviously,
Space
DMC and Gaussian channel are the • *) Binary number (0, 1) together
models for the soft-decision with modulo-2 addition and
demodulation. multiplication form a ILQLWHILHOG or
*DORLV)LHOG Binary Galois field is
• 6RIWGHFLVLRQGHFRGLQJSURYLGHV
VPDOOHU%(5WKDQWKHKDUG
denoted as GF(2).

GHFLVLRQGHFRGLQJ. The finer the Modulo-2 operations in GF(2)

quantization of the demodulator Addition Multiplication
output, the better the BER 0⊕0 = 0 0•0 = 0
performance. Thus, decoding based on 0 ⊕1 = 1 0 •1 = 0
the Gaussian channel provides the 1⊕ 0 = 1 1• 0 = 0
1⊕1 = 0 1•1 = 1
82a 82b
• 3URSHUWLHVRI*) Under e0 = (1,0,0,...,0,0)
addition the LQYHUVH element of 1 is 1 e1 = (0,1,0,...,0,0)
since 1 ⊕ 1 = 0 ; the inverse element of #
0 is 0 since 0 ⊕ 0 = 0 . Under
e n−1 = (0,0,0,...,0,1)
multiplication, the inverse of 1 is also
1 since 1 • 1 = 1. (0’s inverse is not Addition of two vectors in Vn is defined
by modulo-2 addition of corresponding
defined). Subtraction is defined as: u-
elements. For example,
v=u+(-v), where –v is the inverse of v
(1001)+(0101)=(1110).
under addition. Thus 1-1=1+1=0.
• 9HFWRU6SDFH. The set of all binary
Scaling of a vector is defined by the
modulo-2 multiplication. For example,
n-tuples, Vn, is called a vector space 1 • (1001) = (1001)
over GF(2). It is an n-dimensional
0 • (1001) = (0000)
space spanned by n linearly
independent vectors: Distributive law and associative law
apply:
a • (u + v ) = a • u + a • v
( a + b) • v = a • v + b • v
(a • b) • v = a • (b ⊗ • v )
83a 83b
where a, b are constant (1 or 0). combinations of v1 , v 2 ,..., v k forms a k-
dimensional subspace of Vn.
We define the inner product (or dot
product) of two vectors These properties are fundamental for the
u = (u0 , u1 ,..., un−1 ) and v = (v0 , v1 ,..., vn−1 ) algebraic characterization of OLQHDU
as EORFNFRGHVAll codewords of a
u • v = (u0 • v0 , u1 • v1 ,..., un−1 • vn−1 )
linear block code form a subspace.
Inner product also satisfies commutative
7KHLGHDRIFRGLQJis to add (n-k)
and distributive laws.
6XEVSDFHA subset S of vector space redundant bits to messages of k bits, so
Vn is a subspace when the following two that the resultant n-tuples form a
conditions are satisfied: subspace. When received, if any of the
1. The all-zeros vector is in S. codeword is contaminated by noise or
2. The sum of any two vectors in S is interference, it might change into another
also in S (closure property). n-tuple. If it falls outside the subspace, it
can be detected. Of course it might also
Let v1 , v 2 ,..., v k (k<n) be k vectors in a become another codeword, in that case,
vector space Vn. The set of all linear the error cannot be detected.
84a 84b
 Linear Block Codes
Encoding of a linear block code
A linear block code can be generated by
a generator matrix G.

U = mG
where U is the codeword, m is the
uncoded message. For example, a code
generator is
1 1 0 1 0 0 0 
0 1 1 0 1 0 0
G= 
1 1 1 0 0 1 0 
 
 1 0 1 0 0 0 1 
If the message is
m = (1 1 0 1)
then the codeword is
85a 85b
 1 1 0 1 0 0 0 code. In general, a message of k bits is
0 1 1 0 1 0 0 encoded into a codeword of n bits (n>k).
U = (1 1 0 1)  
1 1 1 0 0 1 0
 
1 0 1 0 0 0 1
= (0 0 0 1 1 0 1)
Note that “modulo-2” addition and
multiplication is used in the computation
of the matrix.

Now we see that a message of 4 bits is

encoded into a codeword of 7 bits. We
denote this code as (7, 4) code. Since
there are 24 possible messages, there are
24 codewords. See Table “3.1” for all
possible codewords in the (7, 4) code.
The set of all codewords is called the

86a 86b
The general form of the generator matrix The ratio k/n is called code rate. The
is ratio (n-k)/k is called redundancy. There
are 2k possible codewords corresponding
 V1   v11 v12 " v1n  to all possible k-bit messages. But there
 V  v v " v  are total of 2n possible n-tuples. Among
G =  2  =  21 22 2n 
them only 2k are codewords. From a
#  #  point of view of vector space, the 2n n-
   
 k   k1 k 2
V v v " v kn  tuples form an n-dimensional vectors
space. And the 2k codewords form a k-
where V1, V2, …Vk are OLQHDUO\ dimensional subspace.
LQGHSHQGHQW row vectors (meaning any
Since the codewords form a subspace,
WKHVXPRIDQ\WZRFRGHZRUGVLV
one of them can not be expressed as a

DQRWKHUFRGHZRUGFORVXUHor

linear combination of the others). Thus
U = m1V1 + m2 V2 + ... + mk Vk
That is, a codeword is the linear more generally, a linear combination of
combination of generator vectors. The any number of codewords is still a
weighting numbers are the bits of the codeword. Thus the name “linear” block
message. code. Another important property is that
87a 87b
DOO]HURVQWXSOHPXVWEHD where the 1 is at position n-k+i.
FRGHZRUG in any linear block code. Example:

Systematic Linear Block Codes

A systematic linear block code has a

generator matrix of the following form:

G = [P | I k ] Because of the identity sub-matrix in the

 p11 p12 " p1( n−k ) 1 0 " 0 generator matrix, the bits in the second
p p22 " p2( n−k ) 0 1 " 0 part of the codeword are identical to the
= 
21
message bits, and the first (n-k) bits are
 # # 
  the redundant bits, called parity-check
 pk 1 pk 2 " pk ( n−k ) 0 0 " 1 bits. With systematic generator, the
The ith row is complexity of encoder and the decoder
are reduced. The P and Ik sub-matrices in
g i = ( pi1 , pi 2 ,..., pi ( n−k ) ,0,0,...,1,0,...,0) the generator matrix can exchange

88a 88b
positions without affect the code’s error Parity-Check Matrix
detection and correction capabilities.
If we express the codeword as: The parity-check matrix of a systematic
code is defined as follows.
U = p1 , p2 ,..., pn−k , m1 , m2 ,..., mk H = [I n − k | P T ]
The transpose of H is
Then the relationship between the parity-  1 0 " 0 
check bits and the message bits is as  0 1 " 0 
follows.  
 # 
p1 = m1 p11 + m2 p21 + ... + mk pk 1  
 
I 0 0 " 1
H T =  n -k  =  
p2 = m1 p12 + m2 p22 + ... + mk pk 2  P   p11 p12 " p1( n−k ) 
 
#  p21 p22 " p2( n−k ) 
pn−k = m1 p1( n−k ) + m2 p2( n−k ) + ... + mk pk ( n−k )  # 
 
 pk 1 pk 2 " pk ( n−k ) 
The jth column is
hTj = (0,0,...,1,0,...,0, p1 j , p2 j ,..., pkj )
89a 89b
where 1 is at position j. Thus :KHQ6 ULVDFRGHZRUG
g i <hTj = pij + pij = 0 . RWKHUZLVHULVQRWDFRGHZRUG
And consequently,
GHT = 0 Assuming a codeword U is transmitted
UHT = mGHT = m0 = 0 and when it is received, there are errors
This is a very important property of a in it. These errors can be represented by
codeword. U is a codeword generated by an error pattern e, such that the received
G if and only if UHT = 0. Thus we can n-tuple is r=U+e. For example,
use H to check whether a received n- U=1101000, and r=1100001, then the
tuple is a codeword. error pattern is e=0001001. In fact the
error pattern is the difference between U
Syndrome Test and Error Detection and r, or e = U-r = U+r (modulo-2).
Syndrome is computed as
Denote the received n-tuple as r, let S = rHT = (U + e)HT = eHT
S = rHT Thus the syndrome only depends on the
S is called V\QGURPH. It is a vector of n- error pattern. 7KHPDSSLQJEHWZHHQ
DFRUUHFWDEOHHUURUSDWWHUQDQG
k bits because HT has n-k columns.

90a 90b
DV\QGURPHYHFWRUPXVWEHRQH 1 0 0
WRRQH. This allows the correctable error 0 1 0
 
pattern to be detected and corrected. 0 0 1
 
S = rHT = [1100001] 1 1 0  = [011]
If the error pattern happens to be the
same as a codeword, this error pattern 0 1 1
will produce a zero syndrome. Thus the  
1 1 1
error pattern is not detectable. 1 0 1 
Example: or
1 0 0
0 1 0
1 1 0 1 0 0 0  
0 1 1 0 1 0 0 0 0 1
G=   
1 1 1 0 0 1 0 S = eHT = [0001001] 1 1 0  = [011]
  0 1 1
1 0 1 0 0 0 1
 
1 1 1
U=1101000, r=1100001, e=0001001 1 0 1 
91a 91b
Standard Array and Error Correction called FRVHWOHDGHU, followed by
codewords perturbed by that error
U1 U2 " Ui " U 2k pattern.
e2 U 2 + e2 " Ui + e2 " U 2k + e 2 • The array contains all possible n-
e3 U 2 + e3 " U i + e3 " U 2k + e 3 tuples in the vector space Vn. None is
# # # # missing. None is replicated.
ej U2 + e j " Ui + e j " U 2k + e j • Example: the standard array for a (6,
# # # # 3) code with
e 2n−k U 2 + e 2n−k " U i + e 2n−k " U 2k + e 2n−k 1 0 0
0 1 0
1 1 0 1 0 0   
• The first row is the codewords.   0 0 1
G = 0 1 1 0 1 0 ,H = 
T

• The first column is the correctable   1 1 0
1 0 1 0 0 1  
error patterns. 0 1 1
• U1=e1 is the all-zeros vector which is a  
1 0 1
codeword and also an “error” pattern. The standard array is
• Each row, called a FRVHW, consists of
an error pattern in the first column,
92a 92b
 • The syndrome of a member of a cost is
the same:
S j = (Ui + e j )HT = Ui HT + e j HT

= e j H , ' Ui H = 0
T T

Note that each coset has a unique

syndrome. A syndrome look-up table
• Choosing the error patterns: use the can be prepared for decoding.
error patterns with the least number of
1s first. In the (6, 3) code example, all Error pattern Syndrome
1-bit error patterns are used first. After 000000 000
exhausting all 1-bit patterns, there is 000001 101
still one position left. Then we choose 000010 011
a 2-bit error pattern. This 2-bit pattern 000100 110
must be from those 2-bit patterns that 001000 001
have not appeared in the array. In the 010000 010
example, 010001 is chosen, somewhat 100000 100
arbitrarily. 010001 111
93a 93b
• 'HFRGLQJ3URFHGXUH 1 0 0
0 1 0
1. Calculate the syndrome using  
S = rHT 0 0 1
2. Locate the coset leader ej whose S = rHT = [001110]   = [100]
syndrome equals S. 1 1 0
0 1 1
3. This error pattern is DVVXPHG to be  
the corruption caused by the 1 0 1
channel. From the syndrome table above, the
4. The corrected received vector, or corresponding error pattern is
codeword, is U = r + e j . Adding the eˆ = 100000 . Thus the estimate of the
codeword is
error pattern to the received vector is ˆ = r + eˆ = [001110] + [100000] = [101110]
U
equivalent to subtracting the error
pattern from the received vector. Note that we do not have absolute
• Example: U=101110, r=001110 certainty that the decoder output is what
We calculate the syndrome first. was transmitted since there is possibility
that the received vector is the sum of
another codeword and another
uncorrectable error pattern.
94a 94b
Error-Detecting and Correcting • 0LQLPXPGLVWDQFH of a linear code:
Capability d min = the smallest value of Hamming
distances between any pair of
• +DPPLQJZHLJKW of a vector codewords in the code. If U and V are
w(U) = the number of 1s in the vector, two codewords in a linear code,
e. g., w(110100)=3. W=U+V is also a codeword. Since
• +DPPLQJGLVWDQFH between two d(U, V)=w(U + V)=w(W), the
vectors minimum distance must be equal to
d(U, V) = the number of components minimum weight, i.e, d min = wmin .
in which they differ Since all-zeros vector must be a
e.g., U=110100, V=011010, d(U, V)=4 codeword, the d min is the minimum
• 7KH+DPPLQJGLVWDQFHEHWZHHQ distance between the all-zeros
WZRYHFWRUVHTXDOVWKH+DPPLQJ codeword and all other codewords.
ZHLJKWRIWKHLUYHFWRUVXP • 'HWHUPLQHWKHPLQLPXPGLVWDQFH
d(U, V)=w(U + V) 1. Method 1: Find wmin by inspecting all
e. g., U=110100, V=011010, codewords. This is feasible with
d(U, V)=w(U + V)=w(101110)=4 small-size codes with manual

95a 95b
 inspection. It is also feasible for big- Thus one can inspect the H matrix, find
size codes with computer search. the least number of columns that add to
2. Method 2: use the parity-check zero, this number is equal to d min .
matrix H. 7KHRUHP d min LVHTXDO Again for big codes, computer may be
WRWKHPLQLPXPQXPEHURI programmed to check the H matrix for
OLQHDUO\GHSHQGHQWFROXPQVLQ
the minimum number of linearly
WKH+PDWUL[
dependent columns.
Example, for the (6, 3) code,
Proof: For any codeword 1 0 0 1 0 1 
U = (u1 , u2 ,..., un ) and H = 0 1 0 1 1 0
H = (h1 , h 2 ,..., h n ) , where hi is the ith  
 0 0 1 0 1 1 
column of H, we have
Any two columns add to nonzero. But
UHT = u1h1 + u2h 2 + ... + unh n = 0
columns 1, 2, 4 or 4, 5, 6 add to 0.
If the codeword has the minimum Thus d min = 3 .
• (UURUGHWHFWLRQFDSDELOLW\
weight wmin, then in the above sum
there are wmin columns that add to
zero, i.e., there are wmin columns that A code must detected errors in a
are linearly dependent. codeword before it can be corrected; or
a code may be used for error detection
96a 96b
 only, such as in an ARQ system. The • 0D[LPXPOLNHOLKRRGGHFRGLQJ
error detection capability of a linear Assuming a DMC, when r is the
block code is equal to e = d min − 1 since output vector of the channel (which is
an error pattern of e or less nonzero the output of the demodulator), the
components will not alter the optimal decoder is the PD[LPXP
OLNHOLKRRGGHFRGHU(MLD), which
transmitted codeword to another one.
Such a code also is capable of
computes its likelihood conditioned on
detecting many error patterns with d min
all possible codewords and chooses the
or more errors. In fact it can detect any maximum: decoding in favor of Ui if
error patterns except for those that are P(r | U i ) = max
identical to the 2k codewords. That is, N P (r | U j )
over all U j
the number of detectable error pattern Since the channel is memoryless, the
is 2n-2k. For example, for the (7, 4) likelihood of the vector r is equal to
code, 2n-2k =112. The number of the product of likelihood of all
undetectable error pattern is only 16, a components of the vector. Thus
small fraction of the total error P(r | U) = ∏ P( ri | ui )
patterns. i
or
97a 97b
 log P (r | U ) = ∑ log P (ri | ui ) log(p/(1-p))<0, the smaller the d (r , U ) ,
i the larger the log-likelihood function:
Now consider specializing the MLD log P (r | U ) . Thus the maximum
rule to the BSC. In this case, r is a likelihood-decoding rule becomes
PLQLPXPGLVWDQFHGHFRGLQJ (MDD)
binary sequence that may differ from
the transmitted codeword U in some
rule for a BSC.
• (UURUFRUUHFWLRQFDSDELOLW\
positions. It is clear that
 p, ri ≠ ui
P( ri | ui ) =  See the example in the figure below. U
1 − p, ri = ui and V are two codewords with d min = 5
Thus between them. Using the MDD rule, r1
log P (r | U ) = d (r , U ) log p
and r2 will be decoded as U and r3 will
+[n − d (r , U )]log(1 − p ) be decoded as V. If r1 is the result of a 1-
p bit error to U, the decoder has corrected
= d (r , U ) log+ n log(1 − p )
1− p the error. But if r1 is a result of 4-bit
where d (r , U ) is the Hamming distance error to V, the decoder has made an
error. Similar words can be said for r2
between r and U. For p<1/2, which is
usually true in a practical system, and r3. It is clear that if the error patterns
have more than (dmin-1)/2 errors, the
98a 98b
 where  x  denotes the integer floor of x.
For the (6, 3) code, dmin=3, t=1. However
from the standard array in Fig. “6.11”,
the code can also correct a 2-bit error
pattern. In general, the number of
correctable error patterns is 2n-k since the
standard array has 2n-k error patterns. If
all 2n-k error patterns contain t or fewer
errors, the code is called SHUIHFWFRGH.
The message error probability of a t-
error correcting code is upper-bounded
n
n j
by PM ≤ ∑   p (1 − p ) n− j
j =t +1  j 

and the bit error probability of a t-error

decoder cannot decode correctly. We say correcting code is approximated by
the decoder’s HUURUFRUUHFWLRQ 1 n n j
Pb ≈ ∑ j   p (1 − p ) n− j
FDSDELOLW\ is t = ( d min − 1) / 2  n j =t +1  j 
99a 99b