A COURSE IN PROBABILITY THEORY THIRD EDITIONA COURSE IN PROBABILITY THEORY THIRD EDITION Kai Lai Chung Stanford University ACADEMIC PRESS .nce and Technology Company San Diego San Francisco New York Boston London Sydney TokyoThis book is printed on acid-free paper. @ Copyricht © 2001, 1974, 1968 By ACADEMIC Press ‘ALL RIGHTS RESERVED. [NO PART OF THIS PUBLICATION MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM OR BY ANY MEANS, ELECTRONIC OR MECHANICAL, INCLUDING PHOTOCOPY, RECORDING, OR ANY INFORMATION ‘STORAGE AND RETRIEVAL SYSTEM, WITHOUT PERMISSION IN WRITING FROM THE PUBLISHER. Requests for permission to make copies of any part of the work should be mailed to the following address: Permissions Department, Harcourt, Inc., 6277 Sea Harbor Drive, Orlando, Florida 32887-6777. ACADEMIC PRESS A Harcourt Science and Technology Company 525 B Street, Suite 1900, San Diego, CA 92101-4495, USA hup://www.academicpress.com ACADEMIC PRESS Harcourt Place, 32 Jamestown Road. London, NWI 7BY, UK hutp://www academicpress.com Library of Congress Cataloging in Publication Data; 00-106712 International Standard Book Number: 0-12-174151-6 PRINTED IN THE UNITED STATES OF AMERICA 00 01 02031 987654321Contents Preface to the third edition ix Preface to the second edition xi Preface to the first edition xiii 1 | Distribution function 1.1 Monotone functions 1 1.2 Distribution functions = 7 1.3 Absolutely continuous and singular distributions 11 2 | Measure theory 2.1 Classes of sets. 16 2.2 Probability measures and their distribution functions 21 3 | Random variable. Expectation. Independence 3.1 General definitions 34 3.2 Properties of mathematical expectation 41 3.3 Independence 53 4 | Convergence concepts 4.1 Various modes of convergence 68 4.2 Almost sure convergence; Borel-Cantelli lemma 75vi | CONTENTS 4.3 Vague convergence 84 4.4 Continuation 91 4.5 Uniform integrability; convergence of moments 99 5 | Law of large numbers. Random series 5.1 Simple limit theorems 106 5.2. Weak law of large numbers. 112 5.3 Convergence of series 121 5.4 Strong law of large numbers 129 5.5 Applications 138 Bibliographical Note 148 6 | Characteristic function 6.1 General properties; convolutions 150 6.2 Uniqueness and inversion 160 6.3 Convergence theorems 169 6.4 Simple applications. 175 6.5 Representation theorems 187 6.6 Multidimensional case; Laplace transforms 196 Bibliographical Note 204 7 | Central limit theorem and its ramifications 7.1 Liapounov's theorem — 205 7.2 Lindeberg—Feller theorem 214 7.3. Ramifications of the central limit theorem 224 7.4 Error estimation 235 7.5 Law of the iterated logarithm — 242 7.6 Infinite divisibility 250 Bibliographical Note 261 8 | Random walk 8.1 Zero-or-one laws 263 8.2 Basic notions 270 8.3 Recurrence 278 8.4 Fine structure 288 8.5 Continuation 298 Bibliographical Note 308CONTENTS | vii 9 | Conditioning. Markov property. Martingale 9.1 Basic properties of conditional expectation 310 9.2 Conditional independence; Markov property 322, 9.3 Basic properties of smartingales 334 9.4 Inequalities and convergence 346 9.5 Applications 360 Bibliographical Note 373 | Supplement: Measure and Integral 1 Construction of measure 375 2 Characterization of extensions 380 3 Measures in R = 387 4 Integral 395 5 Applications 407 General Bibliography 413 Index 415Preface to the third edition In this new edition, I have added a Supplement on Measure and Integral. The subject matter is first treated in a general setting pertinent to an abstract measure space, and then specified in the classic Borel-Lebesgue case for the real line. The latter material, an essential part of real analysis, is presupposed in the original edition published in 1968 and revised in the second edition of 1974. When I taught the course under the title “Advanced Probability” at Stanford University beginning in 1962, students from the departments of statistics, operations research (formerly industrial engineering), electrical engi- neering, etc. often had to take a prerequisite course given by other instructors before they enlisted in my course. In later years I prepared a set of notes, lithographed and distributed in the class, to meet the need. This forms the basis of the present Supplement. It is hoped that the result may as well serve in an introductory mode, perhaps also independently for a short course in the stated topics. The presentation is largely self-contained with only a few particular refer- ences to the main text. For instance, after (the old) §2.1 where the basic notions of set theory are explained, the reader can proceed to the first two sections of the Supplement for a full treatment of the construction and completion of a general measure; the next two sections contain a full treatment of the mathe- matical expectation as an integral, of which the properties are recapitulated in §3.2. In the final section, application of the new integral to the older Riemann integral in calculus is described and illustrated with some famous examples. Throughout the exposition, a few side remarks, pedagogic, historical, evenx | PREFACE TO THE THIRD EDITION judgmental, of the kind I used to drop in the classroom, are approximately reproduced. In drafting the Supplement, I consulted Patrick Fitzsimmons on several occasions for support. Giorgio Letta and Bernard Bru gave me encouragement for the uncommon approach to Borel’s lemma in §3, for which the usual proof always left me disconsolate as being too devious for the novice’s appreciation. ‘A small number of additional remarks and exercises have been added to the main text, Warm thanks are due: to Vanessa Gerhard of Academic Press who deci- phered my handwritten manuscript with great ease and care; to Isolde Field of the Mathematics Department for unfailing assistence; to Jim Luce for a mission accomplished. Last and evidently not least, my wife and my daughter Corinna performed numerous tasks indispensable to the undertaking of this publication.Preface to the second edition This edition contains a good number of additions scattered throughout the book as well as numerous voluntary and involuntary changes. The reader who is familiar with the first edition will have the joy (or chagrin) of spotting new entries. Several sections in Chapters 4 and 9 have been rewritten to make the material more adaptable to application in stochastic processes. Let me reiterate that this book was designed as a basic study course prior to various possible specializations. There is enough material in it to cover an academic year in class instruction, if the contents are taken seriously, including the exercises. On the other hand, the ordering of the topics may be varied considerably to suit individual tastes. For instance, Chapters 6 and 7 dealing with limiting distributions can be easily made to precede Chapter 5 which treats almost sure convergence. A specific recommendation is to take up Chapter 9, where conditioning makes a belated appearance, before much of Chapter 5 or even Chapter 4. This would be more in the modem spirit of an early weaning from the independence concept, and could be followed by an excursion into the ‘Markovian territory. Thanks are due to many readers who have told me about errors, obscuri- ties, and inanities in the first edition. An incomplete record includes the names below (with apology for forgotten ones): Geoff Eagleson, Z. Govindarajulu. David Heath, Bruce Henry, Donald Iglehart, Anatole Joffe, Joseph Marker, P. Masani, Warwick Millar, Richard Olshen, S. M. Samuels, David Siegmund, T. Thedéen, A. Gonzélez Villa lobos, Michel Weil, and Ward Whitt. The revised manuscript was checked in large measure by Ditlev Monrad. Thexii | PREFACE TO THE SECOND EDITION galley proofs were read by David Kreps and myself independently, and it was fun to compare scores and see who missed what. But since not all parts of the old text have undergone the same scrutiny, readers of the new edition are cordially invited to continue the fault-finding. Martha Kirtley and Joan Shepard typed portions of the new material. Gail Lemmond took charge of the final page-by-page revamping and it was through her loving care that the revision was completed on schedule. In the third printing a number of misprints and mistakes, mostly minor, are corrected. I am indebted to the following persons for some of these corrections: Roger Alexander, Steven Carchedi, Timothy Green, Joseph Horowitz, Edward Kom, Pierre van Moerbeke, David Siegmund. In the fourth printing, an oversight in the proof of Theorem 6.3.1 is corrected, a hint is added to Exercise 2 in Section 6.4, and a simplification made in (VII) of Section 9.5. A number of minor misprints are also corrected. I am indebted to several readers, including Asmussen, Robert, Schatte, Whitley and Yannaros, who wrote me about the text.