0% found this document useful (0 votes)
4 views133 pagesMaths2
The document is an introduction to error-correcting codes, covering fundamental concepts in digital communication and signal processing. It discusses various types of codes, including linear, cyclic, BCH, and convolutional codes, along with their applications in error detection and correction. The book aims to provide a clear understanding of these techniques for first-time students and professionals in the field, emphasizing the importance of redundancy in information transmission.
Uploaded by
HNCopyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
0% found this document useful (0 votes)
4 views133 pagesMaths2
The document is an introduction to error-correcting codes, covering fundamental concepts in digital communication and signal processing. It discusses various types of codes, including linear, cyclic, BCH, and convolutional codes, along with their applications in error detection and correction. The book aims to provide a clear understanding of these techniques for first-time students and professionals in the field, emphasizing the importance of redundancy in information transmission.
Uploaded by
HNCopyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
/ 133
Introduction to Error-Correcting Codes
Michael Purser
Artech House
Boston e London Library of Congress Cataloging-in-Publication Data
Purser, Michael, 1937-
Introduction to error-correcting codes
Includes bibliographical references and index.
ISBN 0-89006-784-8
1. Signal processing. 2. Error-correcting codes (Information theory) I. Title
TK5102.9.P87 1995 94-32112
621.382°2-de20 CIP
British Library Cataloguing in Publication Data
Purser, Michael
Introduction to Error-correcting Codes
I. Title
621-382
ISBN 0-89006-784-8
© 1995 ARTECH HOUSE, INC.
685 Canton Street
Norwood, MA 02062
All rights reserved. Printed and bound in the United States of America. No part of this book may be
reproduced or utilized in any form or by any means, electronic or mechanical, including photocopy-
ing, recording, or by any information storage and retrieval system, without permission in writing
from the publisher.
International Standard Book Number: 0-89006-784-8
Library of Congress Catalog Card Number: 94-3212,
100987654321 Contents
Preface
Chapter 1 Introduction
1.1 Bit Strings ond Codes
1.1.1 Codes and Error Correction
1.1.2 Erasures and Soft-Decision Decoding
1.2 Hamming Distance and Sphere-Packing
1.3 Shannon’s Theorem
Chapter 2 Linear Codes
2.1 Matrix Representation
2.1.1 The Standard Array
2.1.2. The Generator Matrix
2.2. The Null Matrix or Parity-Check Matrix
2.2.1 The Syndrome
2.2.2. The Columns of the Null Matrix
2.3 Perfect Codes
2.4 Further Bounds on Linear Codes
2.4.1 The Varsharmov-Gilbert Bound
2.4.2. The Plotkin Bound
2.4.3 Bounds in Practice
2.5 Nonbinary Linear Codes
2.5.1. Nonbinary Codes with Characteristic 2
Chapter 3. Cyclic Codes
3.1 The Generating Polynomial
3.1.1 Systematic Cyclic Codes
3.2. The Roots of g(x) and the Null Matrix 33
3.4
3.5
Error Detection with Cyclic Codes
3.3.1 Weight Distributions
3.3.2. Shortened Cyclic Codes and Feedback Shift Registers
Error Correction with Cyclic Codes
Nonbinary Cyclic Codes
Chapter 4 BCH Codes
Minimum Polynomials
The Roots of BCH Codes
Some Examples of BCH Codes
Error Correction of Binary BCH Codes
4.4.1 Practical Procedures for Solving the Equations
4.4.2 An Example of BCH Error Correction
Error Correction of Nonbinary BCH Codes
Reed-Solomon (RS) Codes
4.6.1 A Worked RS Example
4.6.2. An Example of Practical Use of RS Codes
4.6.3 Other Aspects of RS Codes
Chapter 5 Convolutional Codes
5.1
5.2
5.3
5.4
5.5
Tree and Trellis Codes
5.1.1 The Viterbi Algorithm
Linear Convolutional Codes
5.2.1 Control of Decoding Errors
Analysis of Convolutional Codes
Error-Correcting with Convolutional Codes
5.4.1 Soft-Decision Decoding
5.4.2 Sequential Decoding
5.4.3 Feedback Decoding
5.4.4 Syndrome Decoding: A Worked Example
Block Codes and Convolutional Codes
Appendix A Information Theory
Appendix B- Some Binomial Approximations
Appendix C_ Finite Fields
Appendix D The Berlekamp-Massey Algorithm
References
Index Preface
This book aims to be an introduction to the topic of error-correcting codes, a topic
of major importance in digital communication whenever accuracy is critical.
There are several excellent specialist textbooks (and one or two almost
unreadable ones) on the subject, in addition to innumerable articles and papers in
the journals. Despite this, the basics of digital error detection and correction are
unfamiliar to many who might casily understand them, cven when they rely on
those techniques in much of their professional work. In the author’s opinion this
situation arises from the lack of clear, concise introductory texts on the subject;
and it is his hope that this book will he such a text
There are at least three good reasons for studying the subject:
© It is an ingenious and intellectually satisfying discipline in its own right
* It is an application of branches of mathematics, notably that of finite fields,
that gives stimulus to the study of those branches
* It is a technology of immense practical use in computer and telecommunica-
tion systems
This book aims to illuminate all three of these aspects and, by presenting basic
concepts and results, give the reader a firm grasp of the scope of the subject and a
thorough understanding of the principal techniques. On this basis the reader
should be able to pursue any future interest in more specialised areas of the
subject without difficulty. There are many such areas.
The intended readership is the first-time student, whether one who is
formally studying in a third-level institution or a computer or communications
professional wishing to become familiar with the subject by self-education. Mathe-
matical literacy is required, in particular familiarity with linear algebra, but no
further exceptional ability or special knowledge is needed.
The course of the book moves from the general concepts of block codes and
distances (Chapter 1), through linear block codes (Chapter 2), to the special linear
SSeS
rit viii
codes that are cyclic codes (Chapter 3) and the special cyclic codes that are BCH
codes (Chapter 4).
By contrast Chapter 5 is devoted to Convolutional or Trellis Codes, in which
data are in the form of an “infinite” stream, rather than a fixed-length block.
Supporting mathematics is in the Appendices, so that those readers already
familiar with, for example, finite fields can proceed through the main text directly.
The author hopes indeed that his aim of a clear, concise introduction has
been achieved. It remains his pleasure to thank his professional colleagues in the
computer industry and his students at Trinity College in Dublin who helped to
stimulate his interest in error correction and who inspired this book. Special
thanks are due to Jennifer O’Reilly, who typed it. Chapter 1
Introduction
1.1 BIT STRINGS AND CODES
In computers and in digital telecommunications information is almost always
represented in binary form, that is, as a sequence of bits each having the value 0
or 1. In its original form the information may have a different representation. It
could be text represented as words and punctuation based on an alphabet; it could
be speech or video represented as analogue signals. But whatever the original or
intermediate form, the final representation is binary. The alphabet is turned into
6-. 7-, or 8-bit patterns; the speech and video are sampled, quantised, and
digitised.
A sequence of bits may be of any length. For example, digitised TV signals
provide a continuous, virtually endless bit-stream. Alternatively the sequence may
divide naturally into units such as octets or bytes representing the characters in an
alphabet. For practical reasons, sequences of bits are also frequently divided into
blocks that may have little relation to the structure of the content of the informa-
tion. This is done to facilitate handling: a typical example is the blocking of a
bit-stream into “packets” of fixed maximum length (c.g., 256 octets) for transmis-
sion over a packet-switched network.
In this book we are concerned with detecting and correcting errors that
typically occur when such sequences of bits are transmitted over communication
channels, It is worthwhile considering for a moment why one should want to do so,
or at least, why one should want to employ the relatively complex and sophisti-
cated techniques that have been developed for error correction. After all, a large
proportion of distorted information is readily corrected by the human eye, ear, and
brain, without any other aids. One need only consider reading a letter from
someone who cannot spell, or a newspaper, or a corrupted fax message; listening
to an incoherent child or to someone with a speech impediment; or watching a
badly tuned TV set, to realise how great one’s capabilities are for recreating sense
from apparent nonsense. But the essence of this ability to correct errors is that we make use of
redundancy in the information. One or two key words in a garbled message enable
us to guess the rest because the rest was hardly necessary in the first place. A
Spaniard can read an Italian newspaper without knowing Italian, because he
recognises the stems of a few basic nouns and verbs. Most of the rest of the written
text is superfluous to him, Redundancy, often huge redundancy, exists in almost all
information. Consider, for example, ordinary text. There is redundancy
¢ In the orthography. For example, in English the U following a Q is quite
unnecessary. More drastically, it is possible to leave out most vowels from
written English and to reduce all double letters to single ones and still leave
the text intelligible.
* In the syntax. For example, definite and indefinite articles in English can
often be omitted harmlessly; in languages in which the verbs are inflected the
personal pronoun is usually superfluous.
* In the semantics. “At this moment in time” can be cut down to “At this
moment” or “At this time” or indeed discarded in favour of “now.”
(We do not consider complete redundancy in which the message is, for example, a
repetition of what we know already or perhaps just irrelevant.)
But some information is less redundant and more compact than other
information. It is usually impossible to remove a single instruction or parameter
from a computer program in machine language without destroying the program’s
meaning, at least to the computer that interprets it, if not to a skilled and critical
programmer reading it.
From certain viewpoints redundancy is a weakness. It is a waste of space. In
particular it is a waste of capacity on communication channels, which are often
bottlenecks in computer systems if redundant information is transmitted. Many
compression techniques exist for removing redundancy from text and other data
streams; in the case of voice and video these techniques are complex and clever
and compress raw digital signals by factors of 10, or 100, or even more.
A second way of looking at redundancy reveals another associated weakness.
Dssentially redundancy means that out of all possible strings of characters (e.g., in
a text) only a small proportion are valid. Thus, a Q in English not followed by a U
is invalid, and a message containing such a Q is not possible in the normally
written language. This aspect is explored marvellously in Jorge Luis Borges’
“La Biblioteca” (“The Library”), which contains all combinations of letters and
texts possible, and hence contains (in all languages) all knowledge, including, for
example, the future history of the world. The problem is to find the text you are
looking for. If you do find it, is it true? Another text, if you could find it, would tell
you if it is or not (correctly or not?). Perhaps the searching procedure could be
simplified by looking up the catalogue, which must be in the library-if you could
find it. This tantasy conceals an important truth, namely, that in searching for hidden information you use redundancy to distinguish sense from nonsense. In
practical terms if you know the process by which secret text is encrypted and
decrypted without knowing the specific key, trying out all possible keys may enable
you to find the correct one (and hence the secret message) by testing if the
cyphertext decrypted with the test key value makes sense or not. Shannon has
shown that if the key length used in encryption is shorter than the length of the
“information contents” of the secret message this “attack” has a high probability
of success. That is, a trial-and-error attack using all possible keys for decryption
will yield only one meaningful message the correct one. The concepts of ‘“‘informa-
tion content” and “redundancy” as originated by Shannon, the founder of Infor-
mation Theory, are explored in more detail in Appendix A (1,2).
From the point of view of error detection and correction, the custom of
removing redundancy from information either to compress it or to foil “try-all-
the-keys” decryption is a serious problem. Without redundancy one message is as
valid as another, so the corruption of one message necessarily produces another
valid one; and since the recipient has no means of knowing that it has been
corrupted, the error goes undetected. But even with redundancy that enables the
occurrence of an error to be detected by the nonsense that results on re-expanding
the compressed or decrypting the garbled encrypted message, we may be quite
unable to recreate the original data. That is because the complexity of the
expansion and decryption process is likely to spread errors in a small part of the
transmitted data to such an extent that a large part of, if not all the processed data
we corrupted. The error is detected but cannot be corrected. A good example of
this spreading of an error can be scen on many fax machines, which use compres-
sion. and where one or two bit errors may destroy a whole scan line.
(1.1 Codes and Error Correction
lt is the central function of error-control techniques to reintroduce controlled
redundancy, which will overcome these problems, by enabling messages corrupted
in Uransmission to be corrected before further processing. With this controlled
redundancy only a subset of all possible transmitted messages (bit sequences)
contains valid messages. The subset is then called a code, and the valid messages
wwe called codewords or coderectors. A good code is one in which the codewords
are so “separated” that the likelihood of errors corrupting one into another is kept
small,
Error detection then is simplified to answering this question: Is the received
message a codeword or not? If it is a codeword, one assumes that no errors have
occurred. The probability of an undetected error getting through is then the
probability of sufficient errors occurring to transform the real transmitted code-
word into another, apparently correct but in reality a false one. If an error is detected, it can be corrected in principle by one of two
methods:
* The recipient rejects the received message as erroneous and requests the
original transmitter for a repeat transmission (see Figure 1.1). This recovery
by retransmission is commonplace in communication systems where it is
possible. However, if propagation delays, due to distance, are large, the
technique may become so inefficient as to be useless. And then there are
other cases where retransmission is not possible, for example, if one considers
a corrupted archive without a backup.
The recipient corrects the errors by finding the valid codeword “nearest” to
the received message, on the assumption that the nearest is the most likely,
because few corrupting crrors are more likely than many! This procedure is
often called forward error correction (FEC) and is one of the principal topics
of this book. See Figure 1.2.
As an example of these ideas, consider the digits 0 and | extended to 3 bits in
Sender Errors Recipient
Message
oh 1 §
v
Error detected
v
Correct errors
—? To application
Figure 1.1 Error-recovery by detection and retransmission.
Sender Recipient
Errore
Message
sh _
Error detected
Request retransmission
Retransmitted message
—— To application
Figure 1.2 Forward error correction (FEC). length, using 2 redundant bits, to become 010 and 101 respectively. These are then
the only two valid codewords of 3 bits in length; if one receives 000, 001, 011, 100,
110, or 111, one has detected an error.
These invalid vectors can be associated with the valid ones to facilitate error
correction; thus,
010 101
O11 001
110 100
000 «111
where we have placed in the column beneath each codeword those 3-bit patterns
that differ from it by only one bit the “nearest” ones. If we receive O11, for
example, we know it is invalid, but we correct it by saying that it most likely is a
corruption of 010, representing 0. It could, however, be 101, with the first two bits
corrupted, in that case we would have made the error worse by our so-called
correction, but this is a less likely event.
Why is it less likely? The assumption here is that the probability of a
bit being corrupted, p, is small, so that the probability of one error in three,
p (1—p)*, is significantly greater than the probability of two errors in three,
p? (1 —p). If p=0.1, for example, p(1 — p)’ = 0.081, whereas p*(1 — p) = 0.009 (p
is necessarily less than or equal to 0.5. If it were greater, we would simply
complement the entire bit string and then work with (1 — p).)
Obviously an FEC technique that relies on comparing the received message
with all codewords to find the nearest one is impracticable for long codes with
many codewords. For example, many codes will have 21° or 27° codewords.
Such codes are not a more-or-less random collection of reasonably well separated
vectors (as one might be tempted to choose), but rather a carefully structured set
with a complex internal consistency. Error correction then exploits this structure
and consistency to find the codeword nearest to the received vector, using mathe-
matics.
1.1.2. Erasures and Soft-Decision Decoding
We finish this introduction by pointing out that so far we have assumed that the
recipient has been presented with a corrupted received message or vector with no
further information as to the nature and location of the errors. We have also
assumed that the errors, insofar as they affect individual bits, are random and
uniform over all bits. Neither of these assumptions is necessarily true.
In many systems the bits received are the result of some prior processing,
such as demodulation of an analogue waveform, or at least a threshold detector
that decides whether a level is nearer 0 or 1. This preprocessor may qualify the bit values it produces, and this additional information can be used to improve further
the error-correcting procedures. For example, some bits could be flagged as
“uncertain”, that is, either 0 or 1, and could be omitted in the search for the
nearest codeword and filled in only after the codeword was found. A channel that
gives this 3-valued output (0, 1, X = uncertain) is called a Binary Erasure Channel.
To see how an Erasure Channel permits more accurate decoding compared with a
“hard-decision” channel, consider the previous 3-bit example and suppose we
receive OX1. If we had been told that this was 001, we would have said 101 was
transmitted; if we had been told that it was 011, we would have said 010 was
transmitted. But if we work with 0X1, we compare with 0X0 and 1X1, from which
OX1 is equidistant and conclude that we do not know what was transmitted. This is
certainly more accurate (if less conclusive) than “hard-decoding” based on a prior,
possibly arbitrary decision by a preprocessor, of which we are ignorant.
The Erasure Channel is a special example of a more general channel that
delivers qualifying information for each bit, for example, some sort of reliability
factor. These reliability factors can be used to influence the concept and evaluation
of “nearness”. Thus, each bit might have an accompanying factor of value 0.5 to
1.0, which would be the estimated probability that it is what it claims to be. If we
receive 0(0.5), 1(0.6), 1(0.9), where the valuc in parentheses are the probabilities,
then we could say the “distance” to 010 is 0.5 + 0.4 +0.9 = 1.8, whereas the
“distance” to 101 is 0.5 + 0.6 + 0.1 = 1.2, and we would decode the received 011 to
101, not to 010. (Alternatively we could say that the probability of 010 given the
received values is 0.5 - 0.6 - 0.1 = 0.03, whereas the probability of 101 is 0.5 -0.4-0.9
= 0.18 and again choose 101 in preference to 010). This sort of decoding tech-
nique, in which reliability factors are taken into account, is called “soft-decision
decoding”. It has useful applications, particularly in delicate systems, but in many
cases it is simply not practicable, because the receiver of the data is presented with
an uncompromising and unqualified bit stream by other equipment over which he
has little or no control.
The other assumption that has been implicit in this introduction is that bit
errors are random and uniform. In practice this is seldom so. Errors have their
causes, and if that cause is an clectrical “spike”, a momentary short-circuit, or a
scratch on a magnetic medium, it is common that the corruption affects more than
one bit. This will happen if the “noise” lasts for longer than a bit-time or spreads
wider than a bit-space on the medium. Bursts of errors are created, usually at
widely spaced, irregular intervals. Error-correction techniques take account of
burst-errors in various ways, such as
* Rearranging the sequence of the data so that the burst of errors is scattered
randomly throughout it;
* Using specifically designed burst-error-detecting and-correcting codes (c.g.,
Fire codes); © Handling groups of bits, rather than individual bits, as the basic code symbols,
so that a short burst of single-bit errors becomes one symbol error (c.g.,
Reed-Solomon codes).
1.2 HAMMING DISTANCE AND SPHERE-PACKING
Following on from the above heuristic introduction, we may now try to be
somewhat more precise.
We consider block codes. A block code is defined as a subset of all the
possible 2” binary vectors having n bits in length. The distance, also called the
Hamming distance in honour of R.W. Hamming, between two such vectors is
defined as the number of bit positions at which the two vectors have differing
values [3]. The distance of the code, the code-distance, is defined as the minimum
distance between any two members of the code, that is, between any two code-
words.
The first aim in designing a good code is to ensure that its distance is as large
as possible, so as to enable as many errors as possible to be corrected. If the
distance d = 21+ 1, it is clear that we can detect all corruptions that affect <2¢
bits of a codeword, because they can never transform one codeword into another.
It is also clear that we can, at least in principle, correct all corruptions affecting
<1 bits of a codeword, because the corrupted word will be nearer (i.e., have a
smaller distance) to the original codeword than to any other. If d is even, d= 21,
we can detect (2t— 1) errors; but we can correct only (1 — 1) errors, because 1
errors could bring us as near to some other codeword as to the original one. (As an
example, if 00 and 11 are codewords with n = 2, we can detect 01 and 10 as errors
but we cannot correct them).
The second aim in good code design is to construct the code, that is, select
the subset from all possible 2” vectors, in such a way that error detection and
correction can be performed without the need to compare the received vector with
all valid codewords.
Returning to the problem of maximising the distance of the code, it is
obvious that the number of codewords in the code is the principal restriction on
our liberty. If there exist M codewords, we can imagine each codeword sur-
rounded by a “sphere” of related vectors, nearer to it than to any other codeword,
comprising the codeword corrupted by all single-bit errors, all two-bit errors, and
so on, up to all ¢-bit vectors. The number of such related vectors is S(n,t), and
Soe lhe
ll
codeword itself, then each sphere contains (1 + S(n,1)) vectors. 1o respect the ! distance d = 21+ 1 of the code, these spheres cannot overlap; therefore,
M(1+S(n,t)) <2" (1.1)
This is a necessary condition for a code of distance d; but not sufficient.
In Appendix B it is shown that for large n,t an, and a <1/2,1 4 S(n,t) <
2", where H(a) is the entropy function.
H(a) = —(a@log,a+(1—a)log,(1-a)).
Thus, if
M = 2m1-Hoort (2)
is satisfied, (1.1) is certainly satisfied, so we may use (1.2) for large n. For example,
if n =8, tf =2 (so that we have a two-error correcting code, if we can find one),
using inequality (1.1) we get M(1 + 8 + 28) < 256, so M < 6.9.
The codevectors could be 00000000, 01111100, 10011111, and 11100011, but
we have found only four, not six. The code’s (minimum) distance = 2¢ + 1 = 5; the
distance between the second and fourth codewords and the first and third is 6, and
this explains why M is smaller than expected.
Inequality (1.2) is known as the sphere-packing bound. If we take logarithms
to the base 2 in (1.2), we have
log, M ey tly,
A third row is created by picking e, (a vector not already written down) and again
adding it to the first row.
Co c Cy ereeeee Cyt
e ete, ey, $ey...€, + ey_y
es e,te, €, + €7...€, Fly)
This procedure is repeated until no previously selected vectors remain. In the most
recently written row, corresponding to e;, say, no vector e; + ¢,=e, +c, where
j 3. In fact d = 3, because
the rows of G have weight 3, that is, columns 1, 3, 4 and 2, 4, 5 of H are linearly
dependent. All single error patterns are correctable and we can write down the
standard array with the syndromes (the rows of H’) added on the right.
co Cy C2 C3 s
00000 10110 01011 11101 000
10000 00110 11011 01101 110
01000 11110 00011 10101 oll
00100 10010 01111 11001 100
00010 10100 01001 ll 010
Above the broken line is a one-to-one correspondence between the coset leader,
that is, the error pattern in column eg, and the syndrome. Below the broken line,
where we consider 2-error patterns, there is ambiguity. Both 11000 and 00101 give
the same syndrome, 101. We could have chosen 00101 as coset leader, without 18
changing the content of the coset, merely its internal order. If the change were
made we would decode 11000 to 11101 at the head of the column, rather than to
00000. (In the standard array the vectors of weight = 2 have been highlighted.)
2.3 PERFECT CODES
For a given (n—k) and n, (2.7) gives an upper bound for the number of
correctable bits in error, t. An (n, k) linear code for which the inequality becomes
an equality is called a perfect code. The (23, 12) Golay code [5] is an example of a
perfect code, with 1 = 3, d=7, and 1 + 23 + 253 + 1771 = 2048 = 2".
Simpler examples of perfect codes are the Hamming codes [3], with f= 1,
d=3so that l+n=2"*.
For Hamming codes we have then
n (n—k) k
3 2 1
7 3 4
15 4 11
31 5
Note that if 1+n=2"-* then the number of columns of H, 7, is given by
n=2"-*—1, That is, since H has (n — k) rows, all (2"~* — 1) distinct nonzero bit
patterns for the columns of H can be accommodated, giving a distance d= 3,
because no fewer than three columns can be linearly dependent. This situation is
illustrated in the (7,4) Hamming code with
1000101 staple
g— | o1001
= H=|ari0i0
ee 1101001
0001011
Another example of perfect codes is given by the repetition codes, which have
n=2t+1,k=1. With ¢=(n—-1)/2
r+(t)+()+-(s
There is one information bit and (n — 1) check digits. For example, with n =5 the codevectors are (00000) and (11111),
11000
10100
10010
10001
G=(1ll)) H=
and d=5,1=2.
Linear codes whose weight is an odd number can have their distance
increased by adding an extra bit to the code length, n, and setting that bit to 0 or 1
to give all base codewords (i.e., rows of G) even parity (i.e., an even number of 1’s).
This procedure cannot reduce the distance (since the original n bits are unaltered)
but ensures that all codewords have even parity, since linear combinations of
even-parity vectors give more-even-parity vectors. Therefore the minimum weight,
which was odd, must have been increased by at least 1. As an example of this
procedure consider the (7,4) Hamming code (d = 3) extended to an (8,4) code
with d = 4; thus,
10001011 11101000
G—|oloo1i0| 4, _ | o1110100
00101101 11010010
00010111 Wit
G has had parity bits added in an eighth position. H has had zeroes added in the
eighth position so as to leave the existing orthogonality unchanged. H has also had
an all-1’s fourth row added to ensure the even parity of all codewords multiplied
into it. H can be put in more normal form by subtracting the sum of the first three
rows from the last, to give
11101000
01110100
11010010
10110001
2.4 FURTHER BOUNDS ON LINEAR CODES
2.4.1 The Varsharmov-Gilbert Bound
The sphere-packing bound of Inequality (2.7) is attained by perfect codes, of which
the repetition, Hamming, and Golay codes are the only known examples. Given
the number of check digits (m — k), it imposes an upper bound on the distance d,
Or more precisely on t = (d — 1)/2. On the other hand, given (nm — k) we can, as suggested in (2.2), always construct linear codes up to a certain limit by choosing
columns of H to be suitably linearly independent. The limit is given by the
Varsharmov-Gilbert bound [6] as follows:
Suppose we have constructed (n — 1) columns of H subject to the constraint
that all linear combinations of j columns, j < 2t, are linearly independent. Then
we pick a new column x that is not equal to any linear combination of (2t — 1) of
the existing columns. No 2¢ columns of the (now) n columns are linearly depen-
dent; if they were, it would be a contradiction of the assumptions about the first
(n—1) columns and the choice of the nth column. But for this choice to be
possible, in the worst case when all j linear combinations produce distinct values,
we must have
(po page ae (Gt carta a (2.8)
where (2"~* — 1) is the maximum number of nonzero columns.
For large n, (2.8) becomes
Quen DHom) < Q(n-k)
with m =(2t— 1)/(n— 1); see Appendix B. We may rephrase this as follows:
Provided that nH(m) <(n —k), which implies (2.8), we can certainly con-
struct a linear code of distance (2 + 1). For large n this becomes
coderate —k/n <1 H(2¢/n) (2.9)
If (2.9) is satisfied, we can certainly construct an (n,k) linear code capable of
correcting t errors. This should be contrasted with the sphere-packing bound
k/n <1—H(t/n) (2.10)
Inequality (2.10) must be satisfied, since codes that do not satisfy it are impossible;
but if (2.9) is satisfied, we can always find a linear code. The grey area where we
must look for “good” codes with large k/n given t/n, or large t/n given k/n is
1—H(2t/n) 0.
. For the optimum code with maximum codewords
M(n,t) =2* <2d/(2d—n) (if (2d —n) > 0)
Given d, we consider the largest n =n* such that
2d-n*>0, namely, n* = 2d —1
Then M(n*,t) <2d
For larger n we apply the inequality of step 1 repeatedly to get
2* =M(n,t) <2"-" M(n*,1) < 20°19 24
therefore,
ksn+2-2d + logid 0.6
aera)
Code tocar
0.2 Sphore-packing bound
xs
0.08, ot ons. 0.2 0.28 0.3 0.35 8 0.45 as
Figure 2.1 Bounds on coderate and ¢/n for large 1. 4. We conclude that, if nm >n* =2d—1
(n—k) >2d-2-log,d
For large n = 2d — 1 = 41 + 1 we can write this:
coderate =k/n <1—4t/n (2.11)
Inequality (2.11) is the Plotkin bound for linear codes. It is clear from Figure 2.1,
that for high values of t/n this bound is more confining than the sphere-packing
bound.
2.4.3 Bounds in Practice
It is worth looking at some real codes in the light of these various bounds.
Shannon’s theorem states that it is possible to find a code that will correct np
errors, where p is the bit error rate, with arbitrary small probability of error,
provided that the coderate is less than the capacity.
Consider p = 1/7, so that the capacity C = 1 — H(p) = 0.408. The probabil-
ity of decoding failure is the probability that more than ¢ bit errors occur. The
(7.4) Hamming code corrects np = 1 hit error. But the cnderate. k/n = 0.571 (ie.,
in excess of the capacity), and the probability of a decoding failure (i.e.. more than
a single error) is 0.264. We can reduce the coderate to 0.143, by using the (7,1)
repetition code, with ¢=3 and the Prob(Failure) = 0.01. This is certainly much
smaller than 0.264 but not negligible: and the coderate is well below the capacity.
We are nowhere near the theoretical optimum given by Shannon, despite using
perfect” codes, and this is because nis small.
If we try the (15.11) Hamming code as an example of a larger n. with
p=1/15 and C = 0,646, we can still correct np = 1 errors, but k/n = 0.733 (in
excess of C) and the Prob(Failure) = 0.264. Reducing the coderate to 0.067 by
using the (15, 1) repetition code gives a Prob(Failurc) < 5 x 10~°. This is certainly
approaching what one might call “arbitrary small” probability of error, but with a
coderate an order of magnitude less than C.
Table 2.1 presents the (15,7) d= 5, and (15,5) d =7 codes in addition to
those already introduced. In the table are values for the coderate k/n, the ratio
t/n, and the Prob(Failure) assuming the bit-error probability p is 1/7 and 1/15. It
can be seen that only when k/n p in these cases; whereas according to Shannon’s theorem 1/n ~ p can
give negligible Prob(Failure),
In Table 2.1 the Prob(Failure) values corresponding to k/n ...4+q" kl
such columns; therefore
n=(q" *—1)/(q-1)
for a nonbinary Hamming code.
If we consider error correction with such a code, we follow the previous
procedure of evaluating
TH’ =eH’=s
where r is the received vector, e the error vector, and s the syndrome. The scalar 27
product of a column of H will be s if only a single symbol error occurs, and this will
enable us to locate the error postion. To calculate the error value (because it is
not necessarily equal to 1 as is the case for binary codes) we use the parity checks.
For example, suppose c is sent using the (13, 10) code illustrated,
¢ = (0100000000021)
that is, the second row of G, remembering to change the sign as well as transpose
the parity matrix in H. And suppose the error is
e = (0020000000000)
Then r= (0120000000021)
and s=(04+24+04+0,14+04+24+0,2+2+0+1)
= (202)
= 2(101)
therefore, the error is located in the third symbol position, because this is the
position where H” has a row (101). We can conclude directly that the error value
is 2, but it may be more illuminating to do the following: Let x be the correct value
for the position, so c is r with x in the third position.
¢ = (010000000021)
Then, since eH? ~ 0 we find x - 0
The above procedure is typical of nonbinary error correction. Two steps are
used:
1. Locate the position of the symbols in error.
2. Evaluate the values corresponding to those positions.
(There are up to (n—k) equations in principle available for this, from the
requirement cH’ = 0.)
The reasoning that was used (based on cosets of the subspace consisting of
all codewords with a zero in a given position) in (2.1) can be extended to show that
In a code over GF(q) each symbol position has an equal number of all
symbol values, namely g
