Iterative Methods without Inversion 1st Edition

Anatoly Galperin pdf available

Now available at textbookfull.com

https://textbookfull.com/product/iterative-methods-without-
inversion-1st-edition-anatoly-galperin/

★★★★★
4.7 out of 5.0 (59 reviews )

Download PDF Now

 Iterative Methods without Inversion 1st Edition Anatoly
Galperin

TEXTBOOK

Available Formats

■ PDF eBook Study Guide Ebook

EXCLUSIVE 2025 ACADEMIC EDITION – LIMITED RELEASE

Available Instantly Access Library

More products digital (pdf, epub, mobi) instant
download maybe you interests ...

A Contemporary Study of Iterative Methods Convergence

Dynamics and Applications 1st Edition Magrenan

https://textbookfull.com/product/a-contemporary-study-of-
iterative-methods-convergence-dynamics-and-applications-1st-
edition-magrenan/

Iterative methods and their dynamics with applications

: a contemporary study 1st Edition Ioannis Konstantinos
Argyros

https://textbookfull.com/product/iterative-methods-and-their-
dynamics-with-applications-a-contemporary-study-1st-edition-
ioannis-konstantinos-argyros/

Zonal Jets Phenomenology Genesis and Physics Boris

Galperin

https://textbookfull.com/product/zonal-jets-phenomenology-
genesis-and-physics-boris-galperin/

Statistical Inversion of Electromagnetic Logging Data

Qiuyang Shen

https://textbookfull.com/product/statistical-inversion-of-
electromagnetic-logging-data-qiuyang-shen/
An Image Processing Tour of College Mathematics 1st
Edition Yevgeniy V Galperin

https://textbookfull.com/product/an-image-processing-tour-of-
college-mathematics-1st-edition-yevgeniy-v-galperin/

Fluid Dynamics: Part 3 Boundary Layers Anatoly I. Ruban

https://textbookfull.com/product/fluid-dynamics-part-3-boundary-
layers-anatoly-i-ruban/

Academic Collaborations in the Global Marketplace

Anatoly V. Oleksiyenko

https://textbookfull.com/product/academic-collaborations-in-the-
global-marketplace-anatoly-v-oleksiyenko/

Without Choice 1st Edition Elizabeth Andrews

https://textbookfull.com/product/without-choice-1st-edition-
elizabeth-andrews/

Rethinking J S Bach s The Art of Fugue 1st Edition

Anatoly P. Milka

https://textbookfull.com/product/rethinking-j-s-bach-s-the-art-
of-fugue-1st-edition-anatoly-p-milka/
MONOGRAPHS AND RESEARCH NOTES IN MATHEMATICS

Iterative
Methods
without
Inversion

Anatoly Galperin
 MONOGRAPHS AND RESEARCH NOTES IN MATHEMATICS

Series Editors
John A. Burns
Thomas J. Tucker
Miklos Bona
Michael Ruzhansky

Published Titles
Actions and Invariants of Algebraic Groups, Second Edition, Walter Ferrer Santos
and Alvaro Rittatore
Analytical Methods for Kolmogorov Equations, Second Edition, Luca Lorenzi
Application of Fuzzy Logic to Social Choice Theory, John N. Mordeson, Davender S. Malik
and Terry D. Clark
Blow-up Patterns for Higher-Order: Nonlinear Parabolic, Hyperbolic Dispersion and
Schrödinger Equations, Victor A. Galaktionov, Enzo L. Mitidieri, and Stanislav Pohozaev
Complex Analysis: Conformal Inequalities and the Bieberbach Conjecture, Prem K. Kythe
Computational Aspects of Polynomial Identities: Volume l, Kemer’s Theorems, 2nd Edition
Alexei Kanel-Belov, Yakov Karasik, and Louis Halle Rowen
A Concise Introduction to Geometric Numerical Integration, Fernando Casas
and Sergio Blanes
Cremona Groups and Icosahedron, Ivan Cheltsov and Constantin Shramov
Delay Differential Evolutions Subjected to Nonlocal Initial Conditions
Monica-Dana Burlică, Mihai Necula, Daniela Roșu, and Ioan I. Vrabie
Diagram Genus, Generators, and Applications, Alexander Stoimenow
Difference Equations: Theory, Applications and Advanced Topics, Third Edition
Ronald E. Mickens
Dictionary of Inequalities, Second Edition, Peter Bullen
Finite Element Methods for Eigenvalue Problems, Jiguang Sun and Aihui Zhou
Introduction to Abelian Model Structures and Gorenstein Homological Dimensions
Marco A. Pérez
Iterative Methods without Inversion, Anatoly Galperin
Iterative Optimization in Inverse Problems, Charles L. Byrne
Line Integral Methods for Conservative Problems, Luigi Brugnano and Felice Iavernaro
Lineability: The Search for Linearity in Mathematics, Richard M. Aron,
Luis Bernal González, Daniel M. Pellegrino, and Juan B. Seoane Sepúlveda
Modeling and Inverse Problems in the Presence of Uncertainty, H. T. Banks, Shuhua Hu,
and W. Clayton Thompson
Monomial Algebras, Second Edition, Rafael H. Villarreal
Nonlinear Functional Analysis in Banach Spaces and Banach Algebras: Fixed Point
Theory Under Weak Topology for Nonlinear Operators and Block Operator Matrices with
Applications, Aref Jeribi and Bilel Krichen
Published Titles Continued
Partial Differential Equations with Variable Exponents: Variational Methods and Qualitative
Analysis, Vicenţiu D. Rădulescu and Dušan D. Repovš
A Practical Guide to Geometric Regulation for Distributed Parameter Systems
Eugenio Aulisa and David Gilliam
Reconstruction from Integral Data, Victor Palamodov
Signal Processing: A Mathematical Approach, Second Edition, Charles L. Byrne
Sinusoids: Theory and Technological Applications, Prem K. Kythe
Special Integrals of Gradshteyn and Ryzhik: the Proofs – Volume l, Victor H. Moll
Special Integrals of Gradshteyn and Ryzhik: the Proofs – Volume ll, Victor H. Moll
Stochastic Cauchy Problems in Infinite Dimensions: Generalized and Regularized
Solutions, Irina V. Melnikova
Submanifolds and Holonomy, Second Edition, Jürgen Berndt, Sergio Console,
and Carlos Enrique Olmos
The Truth Value Algebra of Type-2 Fuzzy Sets: Order Convolutions of Functions on the
Unit Interval, John Harding, Carol Walker, and Elbert Walker

Forthcoming Titles
Geometric Modeling and Mesh Generation from Scanned Images, Yongjie Zhang
Groups, Designs, and Linear Algebra, Donald L. Kreher
Handbook of the Tutte Polynomial, Joanna Anthony Ellis-Monaghan and Iain Moffat
Microlocal Analysis on Rˆn and on NonCompact Manifolds, Sandro Coriasco
Practical Guide to Geometric Regulation for Distributed Parameter Systems,
Eugenio Aulisa and David S. Gilliam
Symmetry and Quantum Mechanics, Scott Corry
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742

© 2017 by Taylor & Francis Group, LLC

CRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S. Government works

Printed on acid-free paper

Version Date: 20160613

International Standard Book Number-13: 978-1-4987-5892-5 (Hardback)

This book contains information obtained from authentic and highly regarded sources. Reasonable efforts
have been made to publish reliable data and information, but the author and publisher cannot assume
responsibility for the validity of all materials or the consequences of their use. The authors and publishers
have attempted to trace the copyright holders of all material reproduced in this publication and apologize to
copyright holders if permission to publish in this form has not been obtained. If any copyright material has
not been acknowledged please write and let us know so we may rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmit-
ted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented,
including photocopying, microfilming, and recording, or in any information storage or retrieval system,
without written permission from the publishers.

For permission to photocopy or use material electronically from this work, please access www.copyright.
com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood
Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and
registration for a variety of users. For organizations that have been granted a photocopy license by the CCC,
a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used
only for identification and explanation without intent to infringe.

Library of Congress Cataloging‑in‑Publication Data

Names: Galperin, Anatoly.

Title: Iterative methods without inversion / Anatoly Galperin.
Description: Boca Raton, FL : CRC Press, [2016] | Series: Monographs and
research notes in mathematics | Includes bibliographical references and
index.
Identifiers: LCCN 2016025099| ISBN 9781498758925 (hardback : alk. paper) |
ISBN 9781498758963 (e-book)
Subjects: LCSH: Iterative methods (Mathematics) | Numerical analysis. |
Banach spaces. | Hilbert space.
Classification: LCC QA297.8 .G35 2016 | DDC 518/.26--dc23
LC record available at https://lccn.loc.gov/2016025099

Visit the Taylor & Francis Web site at

http://www.taylorandfrancis.com

and the CRC Press Web site at

http://www.crcpress.com
Contents

INTRODUCTION vii

1 Tools of the trade 1

1.1 Banach’s lemma on perturbations . . . . . . . . . . . . . . . 1
1.2 Sherman–Morrison formula . . . . . . . . . . . . . . . . . . 2
1.3 Lemma on sections . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.5 Generalized inversions in Hilbert spaces . . . . . . . . . . . . 5
1.6 Difference equations . . . . . . . . . . . . . . . . . . . . . . 9
1.7 Minimax and maximin . . . . . . . . . . . . . . . . . . . . . 11
1.8 Diagonal operators . . . . . . . . . . . . . . . . . . . . . . . 11

2 Ulm’s method 15
2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 Regular smoothness . . . . . . . . . . . . . . . . . . . . . . . 17
2.3 Majorant generator and convergence lemma . . . . . . . . . 23
2.4 Convergence theorem . . . . . . . . . . . . . . . . . . . . . . 32
2.5 Rate of convergence . . . . . . . . . . . . . . . . . . . . . . . 45
2.6 A posteriori error bounds . . . . . . . . . . . . . . . . . . . . 46
2.7 An application: Chandrasekhar’s integral equation . . . . . . 47
2.8 Research projects . . . . . . . . . . . . . . . . . . . . . . . . 50

3 Ulm’s method without derivatives 53

3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.2 The divided difference operator . . . . . . . . . . . . . . . . 53
3.3 Majorant generator and convergence lemma . . . . . . . . . 64
3.4 Convergence theorem . . . . . . . . . . . . . . . . . . . . . . 84
3.5 Research project . . . . . . . . . . . . . . . . . . . . . . . . . 86

4 Broyden’s method 87
4.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.2 Majorant generator and convergence lemma . . . . . . . . . 88
4.3 Convergence theorem . . . . . . . . . . . . . . . . . . . . . . 100
4.4 Rate of convergence . . . . . . . . . . . . . . . . . . . . . . . 113
4.5 Evaluation of the function f∞ of Proposition 4.5 . . . . . . . 114
4.6 Comparative analysis of iterative methods . . . . . . . . . . 117

v
vi Contents

4.7 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

4.7.1 Complementarity problem. . . . . . . . . . . . . . . . 120
4.7.2 Functional equation . . . . . . . . . . . . . . . . . . . 123
4.7.3 Integral equation . . . . . . . . . . . . . . . . . . . . . 124
4.8 Research projects . . . . . . . . . . . . . . . . . . . . . . . . 125

5 Optimal secant updates of low rank 127

5.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
5.2 Modified Newton method . . . . . . . . . . . . . . . . . . . 128
5.3 Existence and uniqueness of solutions (regular smoothness) . 130
5.4 Modified secant method . . . . . . . . . . . . . . . . . . . . 133
5.5 Existence and uniqueness of solutions (regular continuity). . 139
5.6 Secant updates of low rank . . . . . . . . . . . . . . . . . . . 143
5.7 Optimal secant updates of rank 2 . . . . . . . . . . . . . . . 147
5.8 Optimal on average rank 2 secant updates . . . . . . . . . . 156
5.9 Minimum condition number of I + f(x+ )hv , ·i . . . . . . . . 159
5.10 Research projects . . . . . . . . . . . . . . . . . . . . . . . . 167

6 Optimal secant-type methods 169

6.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
6.2 Existence and uniqueness of solutions (scalar equations) . . . 170
6.3 Optimal methods for scalar equations (Lipschitz smoothness) 173
6.4 Optimal methods for scalar equations (regular smoothness) . 181
6.5 Existence and uniqueness of solutions (Lipschitz continuity of
dd) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
6.6 Research projects . . . . . . . . . . . . . . . . . . . . . . . . 211

7 Majorant generators and their convergence domains 213

7.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
7.2 Convergence domain of the generator (7.1) . . . . . . . . . . 214
7.3 Computation of the convergence domain . . . . . . . . . . . 220
7.4 Research projects . . . . . . . . . . . . . . . . . . . . . . . . 221

References 225

Index 229
INTRODUCTION

This book is about iterative methods for solving nonlinear operator equations
f(x) = 0 , f : X ⊃ D → Y , (0.1)
in Banach or Hilbert spaces. The most widely known (and used) examples of
such methods are Newton’s method
x+ := x − f ′ (x)−1 f(x) (0.2)
and the secant method
x+ := x − [x, x− | f ]−1 f(x) , (0.3)
where the symbol [x, x− | f ] stands for so called divided difference operator
(the reader will find the formal definition of this notion in Chapter 3). How-
ever, these methods and their numerous variants and derivates are left out
of this book because they require inversion of a linear operator at each itera-
tion or at least solution of a linear operator equation. We are exploring only
those methods whose implementation does not involve inversions. One of the
first such methods was proposed by J. Moser [39, 40] in the 1960s. Given a
starting pair (x0 , A0 ), x0 ∈ D, A0 ∈ L(Y, X) (the space of bounded linear op-
erators acting from Y to X), Moser constructed successive iterations (xn , An )
according to the following rule:
x+ := x − Af(x) , A+ := 2A − Af ′ (x)A .
Ulm [58] improved Moser’s method replacing f ′ (x) with f ′ (x+ ). Detailed
analysis of Ulm’s method is the subject of Chapters 2 and 3.
Another example of inversion-free methods is Broyden’s method [4]
Af(x+ )
x+ := x − Af(x) , A+ := A − hA∗ Af(x) , ·i , (0.4)
hA∗ Af(x) , f(x+ ) − f(x)i
where ha , bi is the inner product of two vectors of underlying Hilbert space.
In addition to being inversion-free, it is also derivative-free, which makes it
suitable for solving equations with nondifferentiable operators. This method
is studied in Chapter 4.
Broyden’s method represents the class of iterative methods sometimes
called in literature the secant-update methods. These methods generate the
iterations (xn , An ) according to the rule
x+ := x − Af(x) , A+ := A + B ,

vii
viii INTRODUCTION

where the update B is chosen so that the updated operator A+ satisfies so-
called secant equation

A+ (x+ − x) = f(x+ ) − f(x) . (0.5)

Usually, the update B is a linear operator of low rank (most often 1 or 2),
that has the form ul(·) or u1 l1 (·) + u2 l2 (·) , where u, u1 , u2 are vectors
and l, l1 , l2 are linear functionals. The most widely known example of a
secant-update of rank 2 is BFGS update

y A−1 s
s := x+ −x , y := f(x+ )−f(x) , A−1 −A−1 = hy , ·i− −1 hA−1 s , ·i ,
+
hy , si hA s , si
(0.6)
proposed independently by Broyden [5], Fletcher [10], Goldfarb [25], and
Shanno [53] in 1970 for finite-dimensional unconstrained minimization.
The secant equation (0.5) admits a great variety of solutions A+ for given
vectors x+ − x ∈ X and f(x+ ) − f(x) ∈ Y . This fact inevitably provokes
the question: which one of all solutions (and the corresponding secant-update
methods) is more preferable? The answer to this question depends on a cri-
terion enabling one to compare any two given methods and to decide which
one is better than the other. As such a criterion, we use the entropy of a so-
lution’s position within a set of its guaranteed existence and uniqueness. The
existence of such a set is established by a theorem proved in [21] and recalled
in Chapter 5. The notion of entropy is basic in the theory of information. It is
used to measure the degree of uncertainty of random events in physical sys-
tems, given some (usually incomplete) information about the current system’s
status. The goal of any iterative method designed to solve some problem is to
reduce the uncertainty in a solution’s whereabouts using the information ob-
tained at one iteration. So, it is quite natural to borrow the notion of entropy
to measure the efficiency of iterative methods. A particular representative of
a class of methods for solving operator equations is optimal if its iteration
reduces the uncertainty of a solution’s position (measured by the entropy) as
much as possible for the methods of this class. This optimality criterion was
introduced in [21] and used in [22] to determine the most efficient secant-type
methods. In Chapter 5, the entropy criterion is applied to characterize optimal
secant-updates of rank 2. As it turns out, there are many such updates. So,
one can try to optimize his choice for A+ in (0.5) further. For example, he
may wish to get A+ with the least condition number possible. We show in
Chapter 5 that, in the case of secant-updates of rank 1, this problem has a
nice analytical solution, which leads to a new iterative method.
The next logical step is to analyze the more general class of secant-type
iterative methods,

x+ := x − F x , f(x) ,
which (like the generic secant method (0.3)) require only one evaluation of the
operator f (no derivatives) per iteration. Ulm’s and Broyden’s methods can
 INTRODUCTION ix

be viewed as representatives of the methods of this class. Its members differ

from each other by the mapping F used
to generate the next approximation
x+ from the current iteration x , f(x) . Hence the inevitable question: which
F is the best? In Chapter 6, we try to shed some light on this question in
a one-dimensional setting, using the same entropy optimality criterion as in
Chapter 5.
Prerequisites for reading the book are very modest. It is readable for any-
one with minimal exposure to nonlinear functional analysis. In fact, one feeling
comfortable with the notions of continuity and differentiability of nonlinear
operators acting between Banach and Hilbert spaces should have no difficulty
in understanding developments in the book.
The book is addressed first of all to graduate students and young re-
searchers beginning their career in the field of computational mathematics.
In their interest I suggest several research projects which, in my opinion, are
important for further improvement of the methods discussed in the book.
However, a practitioner also may find something of interest in parts of the
book dealing with examples of the application of those methods to the nu-
merical solution of various infinite-dimensional problems. Having in mind the
needs of this group of potential readers, I make space for details of computer
implementation. My hope is that even an experienced reader will find some
fresh ideas not found elsewhere. I would mention among them regular con-
tinuity, the use of invariants of difference equations in convergence analyses,
and the entropy optimality criterion for iterative methods.
Most (but not all) of the material included in the book has appeared in
journal articles [12]–[24]. However, the process of writing the book has involved
revision of ideas, refinement of some proofs, adding new applications, putting
forward conjectures, and suggesting research directions. The result is that the
exposition in some places deviates considerably from what can be found in
those articles.
Chapter 1
Some useful tools of the trade

This introductory chapter gives some known facts that will be needed later.
The reader can skip it on the first reading and return to it only after being
prompted by a reference.

1.1 Banach’s lemma on perturbations

Lemma 1.1. 1◦ A linear bounded operator A acting from one Banach space
X into another Y is boundedly invertible if and only if

l(A) := min kAxk > 0 . (1.1)

kxk=1

In this case,
1
A−1 = . (1.2)
l(A)
2◦ For every two linear operators A and B ,

l(A) − l(B) ≤ kA − Bk .

Proof. 1◦ Let A be boundedly invertible. If l(A) = 0, then there exists a

sequence xn with kxn k = 1 & kAxn k < 1/n , so that

Axn 1
A−1 := sup A−1 y ≥ A−1 = >n.
kyk=1 kAxn k kAxn k

It follows that A−1 = ∞ , contrary to the hypothesis. Hence, invertibility

of A implies (1.1). Conversely, (1.1) ensures bounded invertibility and (1.2).
Indeed, if A is not invertible, then there can be found a nonzero x0 ∈ X with
x0
Ax0 = 0, so that 0 ≤ l(A) ≤ A = 0, which contradicts the hypothesis
kx0 k
l(A) > 0 . Besides, by the definition of l(A), there exists a sequence xn with
kxn k = 1 & lim kAxn k = l(A). For this sequence, 1 = A−1 Axn ≤ A−1 ·
−1
kAxn k → l(A) A−1 , whence l(A) ≥ A−1 . On the other hand, by the

1
2 Iterative Methods without Inversion

definition of A−1 , ∃ {yn } ⊂ Y with kyn k = 1 & lim A−1 yn = A−1 ,

so that

A−1 yn
1 = AA−1 yn = A · A−1 yn ≥ l(A) A−1 yn → l(A) A−1
A−1 yn

−1
and l(A) ≤ A−1 .
2◦ ∀ x ∈ X with kxk = 1,

kBxk = kAx + (B − A)xk ≥ kAxk − k(B − A)xk ≥ l(A) − kB − Ak.

So, l(B) ≥ l(A) − kB − Ak . By the same reason, l(A) ≥ l(B) − kA − Bk.

Therefore, l(A) − l(B) ≤ kA − Bk.
−1
Corollary 1.2. If A is boundedly invertible and kB − Ak ≤ A−1 ,
then B is boundedly invertible too,

A−1
B−1 ≤ ,
1 − A−1 · kB − Ak

and
2
A−1 kB − Ak
B−1 − A−1 ≤ −1 .
1− A · kB − Ak
−1 −1
Proof. By the lemma, B−1 ≥ A−1 − kB − Ak , and, consequently,

1 A−1
B−1 ≤ −1 = .
A−1 − kB − Ak 1 − A−1 · kB − Ak

As B−1 − A−1 = A−1 (A − B)B−1 ≤ A−1 · B−1 ·kB−Ak, it follows

that
2
A−1 kB − Ak
B−1 − A−1 ≤ .
1 − A−1 · kB − Ak

1.2 Sherman–Morrison formula

Lemma 1.3. Let H and A be a Hilbert space with the inner product h· , ·i
and a linear invertible operator on H. For any two vectors
 Some useful tools of the trade 3

u and v of H, the operator A+ uhv , ·i is invertible if and

only if
hA−1 u , vi 6= −1 ,
in which case
A−1 u
∗
(A + uhv , ·i)−1 = A−1 − A−1 v,· .
1 + A−1 u , v

( T∗ denotes the adjoint of T.)

Proof. The proof is by direct verification of the equalities
!
−1 A−1 u −1 ∗


(A + uhv , ·i) A − A v,· =I
1 + A−1 u , v

and !
−1 A−1 u −1 ∗


A − A v,· (A + uhv , ·i) = I .
1 + A−1 u , v

1.3 Lemma on sections

Lemma 1.4. Let Z and f be a set of pairs (x, y) of arbitrary kind and
a function defined on Z . Denote
 
Y (x) := y (x, y) ∈ Z , X := x Y (x) 6= /❣ .

Then 1◦ Z 6= /❣⇐⇒ X 6= /❣.

2◦ inf f (x, y) = inf inf f (x, y) .
Z x∈X y∈Y (x)
3◦ sup f (x, y) = sup sup f (x, y) .
Z x∈X y∈Y (x)

Proof. 1◦

Z 6= /❣=⇒ ∃ (x0 , y0 ) ∈ Z =⇒ y0 ∈ Y (x0 ) =⇒ Y (x0 ) 6= /❣=⇒ x0 ∈ X

=⇒ X 6= /❣=⇒ ∃ x∗ with Y (x∗ ) 6= /❣=⇒ ∃ y∗ ∈ Y (x∗ )
=⇒ (x∗ , y∗ ) ∈ Z =⇒ Z 6= /❣.

2◦ By definition of inf f (x, y) , ∀ ε > 0 ∃ (xε , yε ) ∈ Z with f (xε , yε ) <

Z
inf f (x, y) + ε. Since yε ∈ Y (xε ), f (xε , yε ) ≥ inf f (xε , y). So,
Z y∈Y (xε )
4 Iterative Methods without Inversion

inf f (xε , y) < inf f (x, y)+ε. Forcing ε to zero results in inf f (xε , y) ≤
y∈Y (xε ) Z y∈Y (xε )
inf f (x, y) . As xε ∈ X,
Z

inf inf f (x, y) ≤ inf f (xε , y) ≤ inf f (x, y). (1.3)

x∈X y∈Y (x) y∈Y (xε ) Z

On the other hand, ∀ ε > 0 ∃ xε ∈ X with inf f (xε , y) <

y∈Y (xε )
inf inf f (x, y) + ε. For all y ∈ Y (xε ) (xε , y) ∈ Z and so
x∈X y∈Y (x)
f (xε , y) ≥ inf f (x, y). Then inf f (xε , y) ≥ inf f (x, y) too. It follows
Z y∈Y (xε ) Z
that inf f (x, y) < inf inf f (x, y) + ε and, consequently, inf f (x, y) ≤
Z x∈X y∈Y (x) Z
inf inf f (x, y). Together with (1.3) this gives the claim.
x∈X y∈Y (x)
◦
3 is proved similarly.
If, in particular, f does not depend on y , then the objective in the
interior extremum is constant, so that inf inf f (x) = inf f (x) and
x∈X y∈Y (x) x∈X
sup sup f (x) = sup f (x) .
x∈X y∈Y (x) x∈X

1.4 Entropy
Entropy is a basic notion of information theory [59] used for measuring
of uncertainty of random events in physical systems. It was introduced by C.
Shennon in his seminal paper [54]. If a system X can be in a finite num-
ber of states x1 , . . . , xn with probabilities (respectively) p1 , . . . , pn , then the
uncertainty of the current status of the system is measured by its entropy
n
X
H(X) := − pi log pi , (1.4)
i=1

where the logarithm can be taken on any base (most often 2,e, or 10): a change
of a base results in multiplying the entropy by a positive constant. If all states
are equiprobable: p1 = . . . = pn = 1/n , then H(X) = log n . For systems
whose possible states are continuously distributed with probability density
f (x), the entropy is defined analogously to (1.4) as
Z ∞
H(X) := − f (x) log f (x) dx .
−∞

In particular, if the states of the system are distributed uniformly on a segment

[a, b] (so that f (x) = 1/(b − a) for x ∈ [a, b] and zero for others), then
Z b
1 1
H(X) = − log dx = log(b − a) ,
a b − a b − a
 Some useful tools of the trade 5

the logarithm of the size of the segment. Because the logarithm is an increasing
function, the size itself can be used as another measure of uncertainty.

1.5 Generalized inversions in Hilbert spaces

It was shown by Penrose [44] that for every (real or complex) matrix A
there exists a unique matrix A† (Moore–Penrose pseudoinverse of A) such
that

∗
∗
AA† A = A & A† AA† = A† & AA† = AA† & A† A = A† A , (1.5)

where the asterisk denotes the Hermite conjugation. Penrose [45] has shown
also that
(a) for any matrix B (of compatible dimensions), the matrix A† B is the
unique minimizer of the Frobenius norm kAX − BkF of minimal
Frobenius norm kXkF and
(b) A† is the unique minimizer of min kXkF AXA = A .

X
The following properties of A† are easily verifiable consequences of (1.5):
(i) if rank(A) is equal to the number of rows of A (the number of
columns), then

−1
−1 ∗

A† = A∗ A∗ A A† = A∗ A A ,

(ii) (A† )† = A ,
(iii) (A∗ )† = (A† )∗ ,

†
∗
†
†
(iv) AA∗ = A† A† , A∗ A = A† A∗ .
Penrose’s result does not hold in general for linear operators between
Hilbert spaces. We can have its analog only for special classes of operators.
Namely, according to one of the corollaries of Theorem 5.1 in [41], the system
of operator equations

AXA = A & XAX = X & (AX)∗ = AX & (XA)∗ = XA . (1.6)

is uniquely solvable for X, if A is bounded and has closed range. Here A∗
denotes the adjoint of A. The solution is called in citeNashed the orthogonal
generalized inverse of A and denoted A† . For example, if A is an orthopro-
jector: A2 = A∗ = A, then A† = A . As another example, consider the (rank
n) operator

Tn : H → En , Tn x := [he1 , xi , . . . , hen , xi] ,

which acts from a Hilbert space H into the Euclidean space En . Here
e1 , . . . , en are orthonormal vectors in H.
6 Iterative Methods without Inversion

Lemma 1.5. 1◦ kTn k = 1.

2◦ The operator
n
X
T†n : En → H , T†n y := yi ei , ∀ y = [y1 , . . . , yn ] ∈ En ,
i=1

is the orthogonal generalized inverse of Tn .

3◦ kT†n k = 1.
n  
4◦ & hei , xi = αi ⇐⇒ x ∈ ni=1 αi ei + I − T†n Tn H
P
i=1 P
n
and i=1 αi ei is the minimum norm solution of the
n
system & hei , xi = αi .
i=1
5◦ The system
n
& hei , xi = αi & kxk = α0 (1.7)
i=1
Pn
is solvable if and only if α0 ≥ 0 & i=1 α2i ≤ α20 .
In this case, it is equivalent to
n
X   n
X
x= αi ei + z & z ∈ I−T†n Tn H & kzk2 = α20 − α2i .
i=1 i=1

Pn Pn
Proof. 1◦ As x = i=1 hei , xiei +x ′ and ′
Pn x = x−2 i=1 hei , xiei is orthogonal
to all ei , it is clear that kxk ≤ 1 =⇒ i=1 hei , xi ≤ 1. So,
n
X
kTn k2 := max kTn xk2 = max k[he1 , xi, . . . , hen , xi]k2 = max hei , xi2 ≤ 1 .
kxk≤1 kxk≤1 kxk≤1
i=1
Pn 2
Pn 2
On the other hand, max i=1 hei , xi ≥ i=1 hei , e1 i = 1.
kxk≤1
2◦ By Theorem 5.1 in [41], it is enough to verify that T†n satisfies the
system
Tn X = Q & XTn = I − P & XTn X = X ,
where P and Q are the orthogonal projectors onto the null space N (Tn ) and
the range R(Tn ) of Tn , respectively:
n
X
P : H → H, Px = x − hei , xiei , Q : En → En , Qy = y,
i=1

(i.e. Q = I, the identity operator on En ). Indeed, ∀ y ∈ En ,

n
! n n
X X X
†
Tn Tn y = Tn yi ei = yi T n ei = yi [he1 , ei i, . . . , hen , ei i]
1 1 1
n
X n
X
= yi [0, . . . , 0, 1i , 0, . . . , 0] = [0, . . . , 0, yi , 0, . . . , 0] = [y1 , . . . , yn ] = y,
1 1
Another Random Scribd Document
with Unrelated Content
Company Wolves sense

and Mr through

in published

the collections

trap been

The

barrels caussis

affairs
change Nostri

the

attend

his government Mr

are civil

the Myers Church

anything
have to

remote lona

new the to

a is

with

and

all the

the space eloquence

and are of

aware

called Royal

principle

ease her on

nihil Ministry the

the provincial

I thoughtful
life

kind of make

theory an 47

a Irish rain

that The of

and is
or If posthumous

which its provinces

been of

the

class he

numbers admirers

and

proved us

liquor the he

confused the
and as name

appears act towns

equal whether trinkets

The

danger s

find opened volumes

Europe those

twelve these shall

from Novels sea

all as 3
for to be

plateau Probus

and not

robes

which endure it

through The

caelis adherents bein

reflected visitor even

wind ready story

and it

in from in

and
not appear

of

one most

history Quotidianis

86
among the does

of propaganda saying

with the this

them half

relates Wherefore

haven defence the

her number

that life
when

An

the quote

last of any

argument that them

both

rooms not Bishops

sufiiciently four doubt

the his

was

deep an records

apologist
making is time

collected are both

his in

a does

that the waterways

Harte end divinity

sancte one boys

law to

to
fill officia Cardinal

Thence M

came should that

in in

and the for

way the
subject followers

sufficiently two which

H end Chinese

the Reply this

to that

Iq

is wonder of

convenient instead
dreams legend great

carved and reported

end

and years landlord

world

wealth Reward clothes

in

woman

no on

of doctrinal
is

it true The

upon is

October

Didst church
of and

that and

upon

of

In that

door

on
sit

desire he it

although chance already

parasites ancient

the by to

The to

Lucas men
meanings

at is

with

apostolic seems If

battle the once

Union and

faith centre it

the even
in

at Fathers

utilitas He upon

True same 3

to as

himself a the

thus must
states from

Meantime the the

assertion

The comes of

as the

in this

horde

our Apparently
describe many

with the

of

outlay excellent fifths

done only

are fountain

sees The Anglican

hearts flowing he
poverty Burcher a

period late

synonyme it

ad speak in

its

been if
it us

will earnest

tourist be obligations

and

handed entire much

in understand

com that

Then If and

other the cotton

to improve he
all river Professor

dies

collecting this

until

superb the

and
shown over

chiefly

risen better

mass the If

there a their

feet

for
the land

position upon in

12

and

of

or sea

to this
critics though and

does

would able left

should

of stop

against as

omni finding
pub forms was

Irish

rich with

not

We direct 1886

The
number interested in

a and Holy

by

foreign

the adoption tract

The is

to a one

of sacrifice owe

mar imprimatur itself

turn Broken

a elements needs

them negotiation evil

onto

among 1834

predecessors
is

with over in

even

encampment sea the

curavit 177

Plot empty iustitiam

ma back

Caspian

pump normally where

in the
offer

the foreign are

his

legislative

Archbishop
you temporal

to

imposing only year

topics

1197 is

PERIPLUS

the 2 enough

78 cheering was

gives theory Us
and

distance He and

us saw

posita the

in works

we

the under source

Bokharian See upon

husband luxuries laymen

there ninety his

actual

hold

the penalties

no

of attitude

legislatures its
length it

stabilibus unfortunately rare

sketching pitiless

queunt

it Review

of

first at We

immediately
the 66 fewness

view

by the for

xxxviii

is

reading and
by

as be

the

and

the

things of form

Still the
its the

were

of

regarded man

we

subordinate genuine an

creature marvellous

used

be

ecclesiastical
much not

and the will

other

have ag organs

to

countries there

F most may

by The of
The true

regard

apparently

by

sovereign Even

preparing

was act
patristic treacherous upper

that wanting

of

representative descending

CeU of
was

who of

of

Lucas

and circularly three

The sea the

contain
on oil

of to

during Eberhard

the laws

encyclical

consistent

to acts in

Christian

Erse
to ut common

done

cutting was

the

generations count the

Episcopi
his

glory

good

Greece

It of

a their historical

the over
point time Cross

from mental

among only rivers

good book

nuns far

anarchy

keeping a and

pursues

They God read

sunshine is the

the inch the

with

using art

Edition the

in
the is votes

shows

susceptam

1884 call

certainly been

this

exile

specialist arms

known
chain

recessed

grieve large additional

with looked deposits

Company
Faith

to

question and enrich

the of

inhabitant

in

to

political
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

textbookfull.com