Introduction to Channel Coding

Introduction to Algebra
EC 515: Coding Theory
Channel coding theorem
Let a DMS with an alphabet X have entropy H(X) and
produce symbols once every T
s
sec. Let a DMC have
capacity C and be used once every T
c
sec. Then, if



there exists a coding scheme for which the source
output can be transmitted over the channel and be
reconstructed with an arbitrarily small prob. of error.
c s
T
C
T
X H
s
) (
Code rate
The parameter C / T
c
is called the critical rate. When
the above expression is satisfied with the equality
sign, the system is said to be signaling at critical rate.
Code rate: a block of k bits mapped to n bits by
adding (n k) redundant bits. Then code rate is given
as r = k / n
Example: The source generates bits 0 or 1 with
equal probability, one bit in T
s
sec. H(X) = 1
The channel is binary symmetric channel with bit-
error probability p, channel carrying one bit in T
c
sec
Code rate
Channel capacity
C = 1 + plog
2
p + (1 p) log
2
(1 p) = 1 H(p)
For every 1/T
s
bits generated per sec, 1/T
c
bits are
transmitted over the channel per sec
Code rate r = T
c
/ T
s .
Then for reliable transmission we require r C.

Channel coding basics
While source coding reduces redundancy, channel
coding introduces redundancy.
Idea is to increase Hamming distance between
every pair of codes; If min. dist. among all pairs of
codes is d
min
and the max. possible bit-error is t, then
Error can be detected if
Error can also be corrected if
One very trivial form of channel coding is repetitive
coding.
1
min
s d t
(

s
2
1
min
d
t
Groups
A set of objects G, along with a binary operation * on
the elements of the set, is called a group if the
following properties are satisfied:
Closure: G is closed under *
Associativity
Guaranteed existence of an identity element.
Existence of an inverse element for every
element.
The group is denoted by {G, * }.
Groups
Theorem: The identity element in a group G is
unique.
Theorem: The inverse of a group element is unique.
Order of a group = number of elements
Finite group = finite order group
Abelian group = if the operation on the set elements
is commutative.
Groups examples
Set of all integers under real addition (abelian)
Set of all rational numbers excluding 0 under real
multiplication (abelian)
Set of binary numbers 0 and 1 under binary operation
XOR (finite, abelian)
Additive group: finite and abelian
(a + b) mod m, where G = {0, 1, 2, ., m1)
Multiplicative group: finite and abelian
(a b) mod p, where G = {1, 2, ., p1), p is prime
Subgroups and cosets
Subgroup: subset H of G if closed under the group
operation * of G and satisfies all conditions of a
group.
Left coset: a * H = {a * h}, where a is an element of
G and h is an element of H
Right coset: H * a
For commutative group, left coset = right coset
Theorem: For any element h in H, its inverse h is
also in H.
Subgroups and cosets
Theorem: No two elements in a coset of H are
identical.
Theorem: No two elements in two different (distinct)
cosets of H are identical.
Every element in G appears in one and only one
coset of H.
All the distinct cosets of H are disjoint.
The union of all the distinct cosets of H forms the
group G.
Fields
A set F together with two binary operations addition
(denoted by +) and multiplication (denoted by ) is
a field if
The set F is a commutative group under addition.
Identity element w.r.t. addition is called additive
identity or zero element (denoted by 0)
The set F minus the zero element is a
commutative group under multiplication.
Identity element w.r.t. multiplication is called
multiplicative identity or unit element (denoted by
1)
Multiplication is distributive over addition.
Fields
Subtracting one field element b from another field
element a = adding additive inverse of b to a
Similarly, dividing a by b = multiplying a with
multiplicative inverse of b
Note, addition, multiplication, subtraction and division
here are field operations and need not be same as
ordinary addition, multiplication, subtraction and
division.
Order of field = number of elements in F.
Finite order field is known as Galois Field.
Field properties
a 0 = 0 a = 0
a b 0, for any two non-zero elements a, b
a b = 0 and a 0 implies b = 0
(a b) = (a) b = a (b)
a b = a c implies b = c, if a 0
Examples of Galois field
Set {0, 1} under mod-2 addition (XOR) and
multiplication (AND)
Called binary field denoted by GF(2)
{0, 1, .., p1} under mod-p addition and mod-p
multiplication, where p is prime.
Called prime field denoted by GF(p)
Extension field: prime field GF(p) extended to a field
of p
m
elements, denoted as GF(p
m
), m is pos. integer
The order of any finite field is a power of a prime.
Binary field arithmetic
We are concerned with binary field GF(2) or its
extension in digital system using binary coding.
Binary mod-2 addition mod-2 subtraction.
Solving equations in binary arithmetic follows the
same rule as in ordinary arithmetic; Cramers rule is
applicable.
Polynomial over GF(2) with one variable and
coefficients 0 or 1:
f(X) = a
0
+ a
1
X + ..+ a
n
X
n
Binary field arithmetic
Degree of polynomial = largest power of X with non-
zero coefficeint, i.e. degree of f(X) is n if a
n
0.
We can have 2
n
polynomials with degree n.
Polynomials over GF(2) can be added, subtracted,
multiplied and divided in the usual way where
multiplication and addition of coefficients are mod-2.
Addition and multiplication of polynomials are
commutative and associative.
Multiplication is distributive over addition.
Euclids division algorithm: f(X) = q(X).g(X) + r(X)
Binary field arithmetic
f(X) divisible by (Xa) (X+a) if a is a root of f(X), i.e.
f(a) = 0.
It follows, f(X) divisible by (X+1) if it has an even
number of terms.
Irreducible polynomial = polynomial p(X) of degree
m not divisible by any polynomial of degree less than
m but greater than 0.
For any m 1, there exists at least one irreducible
polynomial of degree m.
Theorem: Any irreducible polynomial p(X) of
degree m always divides X
n
+ 1, where n = 2
m
1.
Binary field arithmetic
Primitive polynomial: when p(X) does not divide
any other polynomial of the form X
K
+ 1, where K < n
= 2
m
1.
All irreducible polynomials are not primitive
polynomials.
Example: X
4
+ X + 1 and X
4
+ X
3
+ X
2
+ X + 1 are
both irreducible polynomial of degree 4, but only
the first polynomial is also primitive polynomial
(the second polynomial can divide X
5
+ 1)
f
2
(X) = f(X
2
) [f(X)]
2k
= f(X
2k
), k 0.
Construction of GF(2
m
)
Choose a primitive polynomial p(X) over GF(2) of
order m > 1. (Why only a primitive polynomial?)
Determine the root of p(X).
Let, be the root of p(X) p() = 0 and 0,1
since p(X) not divisible by X or (X+1).
Then, F = {0, 1, ,
2
, ,
n1
} forms a Galois field
of order 2
m
= n + 1 under binary field addition + and
multiplication
Multiplication of two non-zero elements of F gives

i

j
=
i+j
(for i, j 0, where
0
= 1), which is
contained in F (in case i + j n, check that
n
= 1)
Construction of GF(2
m
)
Divide every element of F by p() and obtain the
remainder.
Dividing 0 gives remainder 0.
Dividing
i
(0 i < m) gives non-zero remainder
r
i
() =
i
since 0.
Dividing
i
(m i < n) gives remainder r
i
() which
is a non-zero polynomial of degree less than m.
Check that
All r
i
(), for 0 i < n, are distinct.

i
= r
i
() for 0 i < n
Construction of GF(2
m
)
Thus, F = {0, 1, ,
2
, ,
m1
, r
m
(),, r
n1
()} is
the Galois field of order 2
m
.
Check that
Elements of F are distinct polynomials (including
0) of degree less than m.
Number of non-zero polynomials of degree less
than m is n = 2
m
1.
Therefore, adding any two elements of F gives a
polynomial (including 0) of degree less than m
which is also contained in F.
Construction of GF(2
m
)
Thus, GF(2
m
) may be represented in following ways:
Power representation
F = {0, 1, ,
2
, ,
n1
}
Polynomial representation
F = {0, 1, ,
2
, ,
m1
, r
m
(),, r
n1
()}
m-tuple representation: (a
i,0
a
i,1
.. a
i,m1
) where
a
i,k
are the coefficients of the polynomial r
i
()
Example: Construct GF(2
4
) from the generator
polynomial p(X) = 1 + X + X
4

Basic properties of GF(2
m
)
If an element in GF(2
m
) is root of f(X), then
P
, P =
2
l
, l 0 is also root of f(X).
All n = 2
m
1 non-zero elements of GF(2
m
) form all
the roots of X
n
+ 1.
All elements of GF(2
m
), including 0, form all the
roots of X
n+1
+ X.
Minimal polynomial (X) of an element in GF(2
m
)
is the polynomial of smallest degree whose root is ,
i.e. () = 0.
Min. polynomial for 0 and 1 are X and X + 1,
respectively.
Basic properties of GF(2
m
)
Minimal polynomial (X) of an element in GF(2
m
) is
irreducible.
If element in GF(2
m
) is root of f(X), then (X)
divides f(X).
It follows, (X) divides X
n+1
+ X, n = 2
m
1.
If element in GF(2
m
) is root of an irreducible
polynomial f(X), then the minimal polynomial of is
(X) = f(X).
Let, e be the smallest non-negative integer for an
element in GF(2
m
), so that
P
= , P = 2
e
then the
polynomial f(X) given below is irreducible.
Basic properties of GF(2
m
)



The min. polynomial (X) for the element is given
by the polynomial f(X) above.
If the degree of the minimal polynomial (X) for an
element in GF(2
m
) is e then e m, and
P
= ,
where P = 2
e

The element
P
, P = 2
l
is called a conjugate of .

2
, ,
P
, P = 2
e1
form distinct conjugates of .
All conjugates of are all the roots of (X).
Check that
n
= 1
n+1
= e m
( )
[

=
+ =
1
0
2
) (
e
i
i
X X f |
Basic properties of GF(2
m
)
Cyclic group: If there exists an element whose
powers constitute the whole group.
Order of a field element a is the smallest positive
integer N such that a
N
= 1.
Primitive element: If the order of an element in
GF(q) is q 1.
It may be proved that a
q1
= 1 for any element a in
GF(q).
Powers of a primitive element gives all the non-
zero elements of GF(q).
Basic properties of GF(2
m
)
Example: 3 is a primitive element in GF(7), while
2 is not.
If is a primitive element in GF(2
m
) then its
conjugates are also primitives.
Example: For GF(2
4
) = {0, 1, ,
2
, ,
14
},
check that =
7
is a primitive. Accordingly, we
may check that its conjugates
14
,
13
,
11
are also
primitives.
If an element in GF(2
m
) has order N, then all its
conjugates also have order N.
Vector spaces
F is a field whose elements are called scalars.
V is a set of elements on which a binary operation +
is defined.
A multiplication operation between elements of F
and V is defined.
V is called a vector space over the field F if
V is a commutative group under +
a v is in V where a is in F and v in V
Distributive laws for any two elements a, b in F
and any two elements u, v in V.
Vector spaces
Associative laws for any two elements a, b in F
and any element v in V.
1 v = v, where 1 is the unit element in F.
Elements of V are called vectors.
Addition + on V is called vector addition.
Multiplication combining a scalar and a vector is
called scalar multiplication or scalar product.
Additive identity in V is denoted by 0.
Properties of vector spaces
0 v = 0, where 0 is the zero element in F.
a 0 = 0
(a) v = a (v) = (a v)
Therefore, (a) v or a (v) is the additive inverse
of a v
Consider vector space V
n
over GF(2) that has
elements given by n-tuple over GF(2) total 2
n

distinct n-tuples.
Vector 0 is all 0 n-tuple (additive identity)
Vector addition is mod-2 addition.
Additive inverse of an n-tuple is itself.
Properties of vector spaces
Scalar multiplication is mod-2 multiplication.
Subspace: Subset S of V that is also a vector space
over F is called subspace of V.
Let S be a nonempty subset of a vector space V over
a field F. Then, S is a subspace of V if
For any two vectors u, v in S, u + v is also a
vector in S.
For any element a in F and any vector v in S, a v
is also a vector in S.
Properties of vector spaces
Linear combination of vectors v
1
, v
2
, .., v
N
in V:
a
1
v
1
+ a
2
v
2
+ + a
N
v
N
Sum of two linear combinations of v
1
, v
2
, .., v
N
is
also a linear combination of v
1
, v
2
, .., v
N

Product of a scalar in F and a linear combination
of v
1
, v
2
, .., v
N
is also a linear combination of v
1
,
v
2
, .., v
N

Linearly dependent vectors: a
1
v
1
+ + a
N
v
N
= 0
Linearly independent vectors: a
1
v
1
+ + a
N
v
N
0
except when a
1
= a
2
= = a
N
= 0
Basis of vector space
B = A set of linearly independent vectors so that any
vector in V can be expressed as a linear combination
of the vectors in B:
B spans the vector space V and is called the
basis or base of the vector space V.
At least one basis can be found.
Number of basis vectors is the dimension of the
vector space.
Dimension of V
n
= n.
A set of k (where k < n) linearly independent
vectors from V
n
form a k-dimensional subspace of
V
n
consisting of 2
k
vectors.
Dot product
Inner product or dot product: gives scalar result, u
v = u
0
v
0
+ u
1
v
1
+ .. + u
n1
v
n 1
(mod-2
addition)
Orthogonal vectors: u v = 0
Properties of dot product:
u v = v u
u (v + w) = u v + u w
(au) v = a(u v)
Null space
S is a subspace of V
n
.
Define S
d
as another subspace of V
n
so that u v = 0,
where u and v are any vectors from S and S
d
,
respectively.
Check that S
d
must contain vector 0, and so is non-
empty subset of V
n
.
For any element a in GF(2), a v = 0 or v which is
also contained in S
d
.
Also, u (v + w) = 0 where v and w are any two
vectors from S
d

Null space
This means, v and w, and also v + w are
orthogonal to u
Hence, v + w is also in S
d

S
d
is the null or dual space of S
S
d
is the subset of all vectors that are orthogonal to
every vector in S.
Check that, vice-versa is also true.
Hence, conversely, S is the dual space of S
d
Dimension of S
d
is n k.
Example
Let n = 3, k = 2; V
n
is the vector space consisting of
all the 8 number of 3-tuples: 000, 001, 010, 011, 100,
101, 110, 111.
S = {000, 011, 101, 110}
Check that S forms a subspace.
We can find a set of 2 linearly independent vectors
that spans S: {011, 101} or {101, 110} or {011, 110}
2-diemnsional vector space.
Null-space for S is given by S
d
= {000, 111}; vector
111 only spans the vector space S
d
1-dimensional
vector space