Mathematical optimization

“Optimization” and “Optimum” redirect here. For other Given: a function f : A → R from some set A
uses, see Optimization (disambiguation) and Optimum to the real numbers
(disambiguation).
Sought: an element x0 in A such that f(x0 ) ≤
“Mathematical programming” redirects here. For the f(x) for all x in A (“minimization”) or such that
peer-reviewed journal, see Mathematical Programming.
f(x0 ) ≥ f(x) for all x in A (“maximization”).
In mathematics, computer science and operations re-

Such a formulation is called an optimization problem

or a mathematical programming problem (a term not
directly related to computer programming, but still in use
(0,0,4) for example in linear programming – see History below).
Many real-world and theoretical problems may be mod-
5
eled in this general framework. Problems formulated us-
0
ing this technique in the fields of physics and computer vi-
sion may refer to the technique as energy minimization,
z -5
3
speaking of the value of the function f as representing the
-10
2 energy of the system being modeled.
1

-1
0
y Typically, A is some subset of the Euclidean space Rn , of-
-15
-3
-2
-1
0
1
2 -3
-2
ten specified by a set of constraints, equalities or inequali-
3
x
ties that the members of A have to satisfy. The domain A
of f is called the search space or the choice set, while the
elements of A are called candidate solutions or feasible
Graph of a paraboloid given by f(x, y) = −(x² + y²) + 4. The solutions.
global maximum at (0, 0, 4) is indicated by a blue dot.
The function f is called, variously, an objective function,
a loss function or cost function (minimization),[2] a util-
search, mathematical optimization (alternatively, opti- ity function or fitness function (maximization), or, in
mization or mathematical programming) is the selec- certain fields, an energy function or energy functional.
tion of a best element (with regard to some criteria) from A feasible solution that minimizes (or maximizes, if that
some set of available alternatives.[1] is the goal) the objective function is called an optimal so-
In the simplest case, an optimization problem consists of lution.
maximizing or minimizing a real function by systemat- In mathematics, conventional optimization problems are
ically choosing input values from within an allowed set usually stated in terms of minimization. Generally, un-
and computing the value of the function. The generaliza- less both the objective function and the feasible region
tion of optimization theory and techniques to other for- are convex in a minimization problem, there may be sev-
mulations comprises a large area of applied mathematics. eral local minima. A local minimum x* is defined as a
More generally, optimization includes finding “best avail- point for which there exists some δ > 0 so that for all x
able” values of some objective function given a defined such that
domain (or a set of constraints), including a variety of
different types of objective functions and different types
of domains.
∥x − x∗ ∥ ≤ δ;

the expression
1 Optimization problems
f (x∗ ) ≤ f (x)
Main article: Optimization problem
holds; that is to say, on some region around x* all of the
An optimization problem can be represented in the fol- function values are greater than or equal to the value at
lowing way: that point. Local maxima are defined similarly.

1
2 3 HISTORY

A large number of algorithms proposed for solving non- This represents the value (or values) of the argument x in
convex problems – including the majority of commer- the interval (−∞, −1] that minimizes (or minimize) the
cially available solvers – are not capable of making a dis- objective function x2 + 1 (the actual minimum value of
tinction between local optimal solutions and rigorous op- that function is not what the problem asks for). In this
timal solutions, and will treat the former as actual so- case, the answer is x = −1, since x = 0 is infeasible, i.e.
lutions to the original problem. The branch of applied does not belong to the feasible set.
mathematics and numerical analysis that is concerned Similarly,
with the development of deterministic algorithms that are
capable of guaranteeing convergence in finite time to the
actual optimal solution of a non-convex problem is called
arg max x cos(y),
global optimization. x∈[−5,5], y∈R

or equivalently
2 Notation
arg max x cos(y), to: subject x ∈ [−5, 5], y ∈ R,
Optimization problems are often expressed with special x, y
notation. Here are some examples.
represents the (x, y) pair (or pairs) that maximizes (or
maximize) the value of the objective function x cos(y) ,
2.1 Minimum and maximum value of a with the added constraint that x lie in the interval [−5, 5]
function (again, the actual maximum value of the expression does
not matter). In this case, the solutions are the pairs of the
Consider the following notation: form (5, 2kπ) and (−5,(2k+1)π), where k ranges over all
integers.
arg min and arg max are sometimes also written argmin
min (x2 + 1) and argmax, and stand for argument of the minimum
x∈R
and argument of the maximum.
This denotes the minimum value of the objective function
x2 + 1 , when choosing x from the set of real numbers R .
The minimum value in this case is 1 , occurring at x = 0 3 History
.
Similarly, the notation Fermat and Lagrange found calculus-based formulas for
identifying optima, while Newton and Gauss proposed it-
erative methods for moving towards an optimum.
max 2x The term "linear programming" for certain optimization
x∈R
cases was due to George B. Dantzig, although much of
asks for the maximum value of the objective function 2x, the theory had been introduced by Leonid Kantorovich
where x may be any real number. In this case, there is no in 1939. (Programming in this context does not refer to
such maximum as the objective function is unbounded, so computer programming, but from the use of program by
the answer is "infinity" or “undefined”. the United States military to refer to proposed training
and logistics schedules, which were the problems Dantzig
studied at that time.) Dantzig published the Simplex al-
2.2 Optimal input arguments gorithm in 1947, and John von Neumann developed the
theory of duality in the same year.
Main article: Arg max
Other major researchers in mathematical optimization
include the following:
Consider the following notation:
• Aharon Ben-Tal

arg min x2 + 1, • Richard Bellman

x∈(−∞,−1]
• Roger Fletcher
or equivalently • Ronald A. Howard
• Fritz John
arg min x + 1, to: subject x ∈ (−∞, −1].
2
• Narendra Karmarkar
x
 3

• William Karush • Integer programming studies linear programs in

which some or all variables are constrained to take
• Leonid Khachiyan on integer values. This is not convex, and in general
• Bernard Koopman much more difficult than regular linear program-
ming.
• Harold Kuhn
• Quadratic programming allows the objective func-
• László Lovász tion to have quadratic terms, while the feasible set
• Arkadi Nemirovski must be specified with linear equalities and inequal-
ities. For specific forms of the quadratic term, this
• Yurii Nesterov is a type of convex programming.
• Boris Polyak • Fractional programming studies optimization of ra-
tios of two nonlinear functions. The special class of
• Lev Pontryagin concave fractional programs can be transformed to
• James Renegar a convex optimization problem.

• R. Tyrrell Rockafellar • Nonlinear programming studies the general case in

which the objective function or the constraints or
• Cornelis Roos both contain nonlinear parts. This may or may not
• Naum Z. Shor be a convex program. In general, whether the pro-
gram is convex affects the difficulty of solving it.
• Michael J. Todd
• Stochastic programming studies the case in which
• Albert Tucker some of the constraints or parameters depend on
random variables.
• Robust programming is, like stochastic program-
4 Major subfields ming, an attempt to capture uncertainty in the data
underlying the optimization problem. Robust opti-
• Convex programming studies the case when the ob- mization targets to find solutions that are valid under
jective function is convex (minimization) or concave all possible realizations of the uncertainties.
(maximization) and the constraint set is convex.
This can be viewed as a particular case of nonlin- • Combinatorial optimization is concerned with prob-
ear programming or as generalization of linear or lems where the set of feasible solutions is discrete or
convex quadratic programming. can be reduced to a discrete one.

• Linear programming (LP), a type of convex • Stochastic optimization for use with random (noisy)
programming, studies the case in which the function measurements or random inputs in the
objective function f is linear and the con- search process.
straints are specified using only linear equal-
• Infinite-dimensional optimization studies the case
ities and inequalities. Such a set is called a
when the set of feasible solutions is a subset of an
polyhedron or a polytope if it is bounded.
infinite-dimensional space, such as a space of func-
• Second order cone programming (SOCP) is a tions.
convex program, and includes certain types of
quadratic programs. • Heuristics and metaheuristics make few or no as-
• Semidefinite programming (SDP) is a subfield sumptions about the problem being optimized. Usu-
of convex optimization where the underlying ally, heuristics do not guarantee that any optimal so-
variables are semidefinite matrices. It is gen- lution need be found. On the other hand, heuris-
eralization of linear and convex quadratic pro- tics are used to find approximate solutions for many
gramming. complicated optimization problems.
• Conic programming is a general form of con- • Constraint satisfaction studies the case in which the
vex programming. LP, SOCP and SDP can all objective function f is constant (this is used in
be viewed as conic programs with the appro- artificial intelligence, particularly in automated rea-
priate type of cone. soning).
• Geometric programming is a technique • Constraint programming.
whereby objective and inequality constraints
expressed as posynomials and equality con- • Disjunctive programming is used where at least one
straints as monomials can be transformed into constraint must be satisfied but not all. It is of par-
a convex program. ticular use in scheduling.
4 5 CLASSIFICATION OF CRITICAL POINTS AND EXTREMA

In a number of subfields, the techniques are designed pri- 4.2 Multi-modal optimization
marily for optimization in dynamic contexts (that is, de-
cision making over time): Optimization problems are often multi-modal; that is,
they possess multiple good solutions. They could all be
globally good (same cost function value) or there could
• Calculus of variations seeks to optimize an action be a mix of globally good and locally good solutions. Ob-
integral over some space to an extremum by varying taining all (or at least some of) the multiple solutions is
a function of the coordinates the goal of a multi-modal optimizer.
Classical optimization techniques due to their iterative
• Optimal control theory is a generalization of the cal-
approach do not perform satisfactorily when they are used
culus of variations which introduces control policies
to obtain multiple solutions, since it is not guaranteed
that different solutions will be obtained even with dif-
• Dynamic programming studies the case in which the ferent starting points in multiple runs of the algorithm.
optimization strategy is based on splitting the prob- Evolutionary algorithms are however a very popular ap-
lem into smaller subproblems. The equation that de- proach to obtain multiple solutions in a multi-modal op-
scribes the relationship between these subproblems timization task.
is called the Bellman equation.

• Mathematical programming with equilibrium con-

straints is where the constraints include variational 5 Classification of critical points
inequalities or complementarities. and extrema

5.1 Feasibility problem

4.1 Multi-objective optimization
The satisfiability problem, also called the feasibility
Main article: Multi-objective optimization problem, is just the problem of finding any feasible solu-
tion at all without regard to objective value. This can be
Adding more than one objective to an optimization prob- regarded as the special case of mathematical optimization
lem adds complexity. For example, to optimize a struc- where the objective value is the same for every solution,
tural design, one would desire a design that is both light and thus any solution is optimal.
and rigid. When two objectives conflict, a trade-off must Many optimization algorithms need to start from a feasi-
be created. There may be one lightest design, one stiffest ble point. One way to obtain such a point is to relax the
design, and an infinite number of designs that are some feasibility conditions using a slack variable; with enough
compromise of weight and rigidity. The set of trade-off slack, any starting point is feasible. Then, minimize that
designs that cannot be improved upon according to one slack variable until slack is null or negative.
criterion without hurting another criterion is known as
the Pareto set. The curve created plotting weight against
stiffness of the best designs is known as the Pareto fron- 5.2 Existence
tier.
A design is judged to be “Pareto optimal” (equivalently, The extreme value theorem of Karl Weierstrass states that
“Pareto efficient” or in the Pareto set) if it is not dom- a continuous real-valued function on a compact set at-
inated by any other design: If it is worse than another tains its maximum and minimum value. More generally,
design in some respects and no better in any respect, then a lower semi-continuous function on a compact set at-
it is dominated and is not Pareto optimal. tains its minimum; an upper semi-continuous function on
a compact set attains its maximum.
The choice among “Pareto optimal” solutions to deter-
mine the “favorite solution” is delegated to the decision
maker. In other words, defining the problem as multi- 5.3 Necessary conditions for optimality
objective optimization signals that some information is
missing: desirable objectives are given but not their de- One of Fermat’s theorems states that optima of uncon-
tailed combination. In some cases, the missing informa- strained problems are found at stationary points, where
tion can be derived by interactive sessions with the deci- the first derivative or the gradient of the objective func-
sion maker. tion is zero (see first derivative test). More generally, they
Multi-objective optimization problems have been gener- may be found at critical points, where the first derivative
alized further into vector optimization problems where or gradient of the objective function is zero or is unde-
the (partial) ordering is no longer given by the Pareto or- fined, or on the boundary of the choice set. An equa-
dering. tion (or set of equations) stating that the first derivative(s)
 5

equal(s) zero at an interior optimum is called a 'first-order the point is a local maximum; finally, if indefinite, then
condition' or a set of first-order conditions. the point is some kind of saddle point.
Optima of equality-constrained problems can be found by Constrained problems can often be transformed into un-
the Lagrange multiplier method. The optima of problems constrained problems with the help of Lagrange multipli-
with equality and/or inequality constraints can be found ers. Lagrangian relaxation can also provide approximate
using the 'Karush–Kuhn–Tucker conditions'. solutions to difficult constrained problems.
When the objective function is convex, then any local
minimum will also be a global minimum. There exist ef-
5.4 Sufficient conditions for optimality ficient numerical techniques for minimizing convex func-
tions, such as interior-point methods.
While the first derivative test identifies points that might
be extrema, this test does not distinguish a point that is a
minimum from one that is a maximum or one that is nei-
ther. When the objective function is twice differentiable, 6 Computational optimization
these cases can be distinguished by checking the second techniques
derivative or the matrix of second derivatives (called the
Hessian matrix) in unconstrained problems, or the ma-
To solve problems, researchers may use algorithms that
trix of second derivatives of the objective function and
terminate in a finite number of steps, or iterative meth-
the constraints called the bordered Hessian in constrained
ods that converge to a solution (on some specified class
problems. The conditions that distinguish maxima, or
of problems), or heuristics that may provide approximate
minima, from other stationary points are called 'second-
solutions to some problems (although their iterates need
order conditions’ (see 'Second derivative test'). If a candi-
not converge).
date solution satisfies the first-order conditions, then satis-
faction of the second-order conditions as well is sufficient
to establish at least local optimality. 6.1 Optimization algorithms
• Simplex algorithm of George Dantzig, designed for
5.5 Sensitivity and continuity of optima linear programming.
The envelope theorem describes how the value of an • Extensions of the simplex algorithm, designed for
optimal solution changes when an underlying parameter quadratic programming and for linear-fractional
changes. The process of computing this change is called programming.
comparative statics.
• Variants of the simplex algorithm that are especially
The maximum theorem of Claude Berge (1963) de- suited for network optimization.
scribes the continuity of an optimal solution as a function
of underlying parameters. • Combinatorial algorithms

5.6 Calculus of optimization 6.2 Iterative methods

Main article: Karush–Kuhn–Tucker conditions Main article: Iterative method
See also: Critical point (mathematics), Differential See also: Newton’s method in optimization, Quasi-
calculus, Gradient, Hessian matrix, Positive definite Newton method, Finite difference, Approximation
matrix, Lipschitz continuity, Rademacher’s theorem, theory, and Numerical analysis
Convex function, and Convex analysis
The iterative methods used to solve problems of nonlinear
For unconstrained problems with twice-differentiable programming differ according to whether they evaluate
functions, some critical points can be found by finding Hessians, gradients, or only function values. While eval-
the points where the gradient of the objective function uating Hessians (H) and gradients (G) improves the rate
is zero (that is, the stationary points). More generally, a of convergence, for functions for which these quantities
zero subgradient certifies that a local minimum has been exist and vary sufficiently smoothly, such evaluations in-
found for minimization problems with convex functions crease the computational complexity (or computational
and other locally Lipschitz functions. cost) of each iteration. In some cases, the computational
Further, critical points can be classified using the complexity may be excessively high.
definiteness of the Hessian matrix: If the Hessian is pos- One major criterion for optimizers is just the number
itive definite at a critical point, then the point is a local of required function evaluations as this often is already
minimum; if the Hessian matrix is negative definite, then a large computational effort, usually much more effort
6 6 COMPUTATIONAL OPTIMIZATION TECHNIQUES

than within the optimizer itself, which mainly has to oper- • Ellipsoid method: An iterative method for
ate over the N variables. The derivatives provide detailed small problems with quasiconvex objective
information for such optimizers, but are even harder to functions and of great theoretical interest, par-
calculate, e.g. approximating the gradient takes at least ticularly in establishing the polynomial time
N+1 function evaluations. For approximations of the 2nd complexity of some combinatorial optimiza-
derivatives (collected in the Hessian matrix) the number tion problems. It has similarities with Quasi-
of function evaluations is in the order of N². Newton’s Newton methods.
method requires the 2nd order derivates, so for each it-
• Reduced gradient method (Frank–Wolfe) for
eration the number of function calls is in the order of
approximate minimization of specially struc-
N², but for a simpler pure gradient optimizer it is only
tured problems with linear constraints, espe-
N. However, gradient optimizers need usually more iter-
cially with traffic networks. For general un-
ations than Newton’s algorithm. Which one is best with
constrained problems, this method reduces to
respect to the number of function calls depends on the
the gradient method, which is regarded as ob-
problem itself.
solete (for almost all problems).
• Simultaneous perturbation stochastic approx-
• Methods that evaluate Hessians (or approximate
imation (SPSA) method for stochastic opti-
Hessians, using finite differences):
mization; uses random (efficient) gradient ap-
• Newton’s method proximation.
• Sequential quadratic programming: A • Methods that evaluate only function values: If a
Newton-based method for small-medium problem is continuously differentiable, then gradi-
scale constrained problems. Some ver- ents can be approximated using finite differences,
sions can handle large-dimensional prob- in which case a gradient-based method can be used.
lems.
• Interpolation methods
• Methods that evaluate gradients or approximate gra-
dients using finite differences (or even subgradi- • Pattern search methods, which have better
ents): convergence properties than the Nelder–Mead
heuristic (with simplices), which is listed be-
• Quasi-Newton methods: Iterative methods for low.
medium-large problems (e.g. N<1000).
• Conjugate gradient methods: Iterative meth-
ods for large problems. (In theory, these 6.3 Global convergence
methods terminate in a finite number of steps
with quadratic objective functions, but this fi- More generally, if the objective function is not a
nite termination is not observed in practice on quadratic function, then many optimization methods use
finite–precision computers.) other methods to ensure that some subsequence of itera-
• Interior point methods: This is a large class of tions converges to an optimal solution. The first and still
methods for constrained optimization. Some popular method for ensuring convergence relies on line
interior-point methods use only (sub)gradient searches, which optimize a function along one dimen-
information, and others of which require the sion. A second and increasingly popular method for en-
evaluation of Hessians. suring convergence uses trust regions. Both line searches
and trust regions are used in modern methods of non-
• Gradient descent (alternatively, “steepest de- differentiable optimization. Usually a global optimizer
scent” or “steepest ascent”): A (slow) method is much slower than advanced local optimizers (such as
of historical and theoretical interest, which has BFGS), so often an efficient global optimizer can be con-
had renewed interest for finding approximate structed by starting the local optimizer from different
solutions of enormous problems. starting points.
• Subgradient methods - An iterative method
for large locally Lipschitz functions using
generalized gradients. Following Boris T. 6.4 Heuristics
Polyak, subgradient–projection methods are
similar to conjugate–gradient methods. Main article: Heuristic algorithm
• Bundle method of descent: An iterative
method for small–medium-sized problems Besides (finitely terminating) algorithms and (conver-
with locally Lipschitz functions, particularly gent) iterative methods, there are heuristics that can pro-
for convex minimization problems. (Similar vide approximate solutions to some optimization prob-
to conjugate gradient methods) lems:
7.3 Electrical engineering 7

• Memetic algorithm with game theory and the study of economic equilibria.
The Journal of Economic Literature codes classify math-
• Differential evolution ematical programming, optimization techniques, and re-
• Evolutionary algorithms lated topics under JEL:C61-C63.
In microeconomics, the utility maximization problem and
• Dynamic relaxation its dual problem, the expenditure minimization problem,
• Genetic algorithms are economic optimization problems. Insofar as they be-
have consistently, consumers are assumed to maximize
• Hill climbing with random restart their utility, while firms are usually assumed to maximize
their profit. Also, agents are often modeled as being risk-
• Nelder-Mead simplicial heuristic: A popular heuris- averse, thereby preferring to avoid risk. Asset prices are
tic for approximate minimization (without calling also modeled using optimization theory, though the un-
gradients) derlying mathematics relies on optimizing stochastic pro-
cesses rather than on static optimization. Trade theory
• Particle swarm optimization
also uses optimization to explain trade patterns between
• Artificial bee colony optimization nations. The optimization of market portfolios is an ex-
ample of multi-objective optimization in economics.
• Simulated annealing
Since the 1970s, economists have modeled dynamic de-
• Tabu search cisions over time using control theory. For example, mi-
croeconomists use dynamic search models to study labor-
• Reactive Search Optimization (RSO)[3] imple- market behavior.[5] A crucial distinction is between deter-
mented in LIONsolver ministic and stochastic models.[6] Macroeconomists build
dynamic stochastic general equilibrium (DSGE) mod-
els that describe the dynamics of the whole economy as
7 Applications the result of the interdependent optimizing decisions of
workers, consumers, investors, and governments.[7][8]
7.1 Mechanics
7.3 Electrical engineering
Problems in rigid body dynamics (in particular articu-
lated rigid body dynamics) often require mathematical
Some common applications of optimization techniques
programming techniques, since you can view rigid body
in electrical engineering include active filter design,[9]
dynamics as attempting to solve an ordinary differential
stray field reduction in superconducting magnetic energy
equation on a constraint manifold; the constraints are var-
storage systems, space mapping design of microwave
ious nonlinear geometric constraints such as “these two
structures,[10] handset antennas, electromagnetic design.
points must always coincide”, “this surface must not pene-
trate any other”, or “this point must always lie somewhere
on this curve”. Also, the problem of computing contact 7.4 Operations research
forces can be done by solving a linear complementarity
problem, which can also be viewed as a QP (quadratic Another field that uses optimization techniques exten-
programming) problem. sively is operations research.[11] Operations research also
Many design problems can also be expressed as optimiza- uses stochastic modeling and simulation to support im-
tion programs. This application is called design opti- proved decision-making. Increasingly, operations re-
mization. One subset is the engineering optimization, search uses stochastic programming to model dynamic
and another recent and growing subset of this field is decisions that adapt to events; such problems can be
multidisciplinary design optimization, which, while use- solved with large-scale optimization and stochastic opti-
ful in many problems, has in particular been applied to mization methods.
aerospace engineering problems.

7.5 Control engineering

7.2 Economics
Mathematical optimization is used in much modern con-
Economics is closely enough linked to optimization of troller design. High-level controllers such as model pre-
agents that an influential definition relatedly describes dictive control (MPC) or real-time optimization (RTO)
economics qua science as the “study of human behav- employ mathematical optimization. These algorithms
ior as a relationship between ends and scarce means” run online and repeatedly determine values for decision
with alternative uses.[4] Modern optimization theory in- variables, such as choke openings in a process plant, by
cludes traditional optimization theory but also overlaps iteratively solving a mathematical optimization problem
8 11 FURTHER READING

including constraints and a model of the system to be con- [7] Julio Rotemberg and Michael Woodford (1997), “An
trolled. Optimization-based Econometric Framework for the
Evaluation of Monetary Policy.NBER Macroeconomics
Annual, 12, pp. 297-346.
7.6 Hydrology [8] From The New Palgrave Dictionary of Economics (2008),
2nd Edition with Abstract links:
Nonlinear optimization methods are applied to generate • "numerical optimization methods in economics" by Karl
computational models of subsurface reservoirs (such as Schmedders
water resources or oil reservoirs).[12] • "convex programming" by Lawrence E. Blume
• "Arrow–Debreu model of general equilibrium" by John
Geanakoplos.
7.7 Geophysics [9] Bishnu Prasad De, R. Kar, D. Mandal, S. P. Ghoshal,
“Optimal selection of components value for analog ac-
Optimization techniques are regularly used in geophysical tive filter design using simplex particle swarm optimiza-
parameter estimation problems. Given a set of geophysi- tion”, International Journal of Machine Learning and Cy-
cal measurements, e.g. seismic recordings, it is common bernetics August 2015, Volume 6, Issue 4, pp 621–636,
to solve for the physical properties and geometrical shapes 27 September 2014.
of the underlying rocks and fluids. [10] S. Koziel and J.W. Bandler, “Space mapping with multi-
ple coarse models for optimization of microwave compo-
nents,” IEEE Microwave and Wireless Components Let-
7.8 Molecular modeling ters, vol. 8, no. 1, pp. 1–3, Jan. 2008.
[11] “New force on the political scene: the Seophonisten”.
Main article: Molecular modeling http://www.seophonist-wahl.de. Retrieved 14 September
2013. External link in |publisher= (help)
Nonlinear optimization methods are widely used in [12] “History matching production data and uncertainty as-
conformational analysis. sessment with an efficient TSVD parameterization algo-
rithm”. Journal of Petroleum Science and Engineering
113: 54–71. doi:10.1016/j.petrol.2013.11.025.
8 Solvers
Main article: List of optimization software 11 Further reading

11.1 Comprehensive

9 See also 11.1.1 Undergraduate level

• Bradley, S.; Hax, A.; Magnanti, T. (1977). Applied

10 Notes mathematical programming. Addison Wesley.
• Rardin, Ronald L. (1997). Optimization in opera-
[1] "The Nature of Mathematical Programming,” Mathemat- tions research. Prentice Hall. p. 919. ISBN 0-02-
ical Programming Glossary, INFORMS Computing Soci- 398415-5. copyright: 1998
ety.
• Strang, Gilbert (1986). Introduction to applied
[2] W. Erwin Diewert (2008). “cost functions,” The New Pal- mathematics. Wellesley, MA: Wellesley-Cambridge
grave Dictionary of Economics, 2nd Edition Contents. Press (Strang’s publishing company). pp. xii+758.
ISBN 0-9614088-0-4. MR 870634.
[3] Battiti, Roberto; Mauro Brunato; Franco Mascia (2008).
Reactive Search and Intelligent Optimization. Springer
Verlag. ISBN 978-0-387-09623-0. 11.1.2 Graduate level

[4] Lionel Robbins (1935, 2nd ed.) An Essay on the Nature • Magnanti, Thomas L. (1989). “Twenty years of
and Significance of Economic Science, Macmillan, p. 16. mathematical programming”. In Cornet, Bernard;
Tulkens, Henry. Contributions to Operations Re-
[5] A. K. Dixit ([1976] 1990). Optimization in Economic The-
search and Economics: The twentieth anniversary of
ory, 2nd ed., Oxford. Description and contents preview.
CORE (Papers from the symposium held in Louvain-
[6] A.G. Malliaris (2008). “stochastic optimal control,” The la-Neuve, January 1987). Cambridge, MA: MIT
New Palgrave Dictionary of Economics, 2nd Edition. Press. pp. 163–227. ISBN 0-262-03149-3. MR
Abstract. 1104662.
11.3 Combinatorial optimization 9

• Minoux, M. (1986). Mathematical programming: • P. E. Gill, W. Murray and M. H. Wright (1982).

Theory and algorithms. Egon Balas foreword) Practical Optimization. Emerald Publishing. ISBN
(Translated by Steven Vajda from the (1983 Paris: 978-0-12-283952-8.
Dunod) French ed.). Chichester: A Wiley-
Interscience Publication. John Wiley & Sons, Ltd. • Xin-She Yang (2010). Engineering Optimization:
pp. xxviii+489. ISBN 0-471-90170-9. MR An Introduction with Metaheuristic Applications.
2571910. (2008 Second ed., in French: Program- Wiley. ISBN 978-0-470-58246-6.
mation mathématique: Théorie et algorithmes. Edi- • Bonnans, J. Frédéric; Gilbert, J. Charles;
tions Tec & Doc, Paris, 2008. xxx+711 pp. ISBN Lemaréchal, Claude; Sagastizábal, Claudia A.
978-2-7430-1000-3. MR 868279. (2006). Numerical optimization: Theoretical and
• Nemhauser, G. L.; Rinnooy Kan, A. H. G.; Todd, practical aspects. Universitext (Second revised
M. J., eds. (1989). Optimization. Handbooks in ed. of translation of 1997 French ed.). Berlin:
Operations Research and Management Science 1. Springer-Verlag. pp. xiv+490. doi:10.1007/978-
Amsterdam: North-Holland Publishing Co. pp. 3-540-35447-5. ISBN 3-540-35445-X. MR
xiv+709. ISBN 0-444-87284-1. MR 1105099. 2265882.

• J. E. Dennis, Jr. and Robert B. Schnabel, A • Bonnans, J. Frédéric; Shapiro, Alexander (2000).
view of unconstrained optimization (pp. 1– Perturbation analysis of optimization problems.
72); Springer Series in Operations Research. New York:
• Donald Goldfarb and Michael J. Todd, Linear Springer-Verlag. pp. xviii+601. ISBN 0-387-
programming (pp. 73–170); 98705-3. MR 1756264.
• Philip E. Gill, Walter Murray, Michael A. • Boyd, Stephen P.; Vandenberghe, Lieven (2004).
Saunders, and Margaret H. Wright, Con- Convex Optimization (pdf). Cambridge University
strained nonlinear programming (pp. 171– Press. ISBN 978-0-521-83378-3. Retrieved Octo-
210); ber 15, 2011.
• Ravindra K. Ahuja, Thomas L. Magnanti, and
• Jorge Nocedal and Stephen J. Wright (2006).
James B. Orlin, Network flows (pp. 211–369);
Numerical Optimization. Springer. ISBN 0-387-
• W. R. Pulleyblank, Polyhedral combinatorics 30303-0.
(pp. 371–446);
• George L. Nemhauser and Laurence A. • Ruszczyński, Andrzej (2006). Nonlinear Optimiza-
Wolsey, Integer programming (pp. 447–527); tion. Princeton, NJ: Princeton University Press. pp.
xii+454. ISBN 978-0-691-11915-1. MR 2199043.
• Claude Lemaréchal, Nondifferentiable opti-
mization (pp. 529–572); • Robert J. Vanderbei (2013). Linear Programming:
• Roger J-B Wets, Stochastic programming (pp. Foundations and Extensions, 4th Edition. Springer.
573–629); ISBN 978-1-4614-7629-0.
• A. H. G. Rinnooy Kan and G. T. Timmer,
Global optimization (pp. 631–662); 11.3 Combinatorial optimization
• P. L. Yu, Multiple criteria decision making:
five basic concepts (pp. 663–699). • R. K. Ahuja, Thomas L. Magnanti, and James B.
Orlin (1993). Network Flows: Theory, Algorithms,
• Shapiro, Jeremy F. (1979). Mathematical program-
and Applications. Prentice-Hall, Inc. ISBN 0-13-
ming: Structures and algorithms. New York: Wiley-
617549-X.
Interscience [John Wiley & Sons]. pp. xvi+388.
ISBN 0-471-77886-9. MR 544669. • William J. Cook, William H. Cunningham, William
R. Pulleyblank, Alexander Schrijver; Combinato-
• Spall, J. C. (2003), Introduction to Stochastic Search
rial Optimization; John Wiley & Sons; 1 edition
and Optimization: Estimation, Simulation, and Con-
(November 12, 1997); ISBN 0-471-55894-X.
trol, Wiley, Hoboken, NJ.
• Gondran, Michel; Minoux, Michel (1984). Graphs
and algorithms. Wiley-Interscience Series in Dis-
11.2 Continuous optimization crete Mathematics (Translated by Steven Vajda
from the second (Collection de la Direction des
• Roger Fletcher (2000). Practical methods of opti-
Études et Recherches d'Électricité de France [Collec-
mization. Wiley. ISBN 978-0-471-49463-8.
tion of the Department of Studies and Research of
• Mordecai Avriel (2003). Nonlinear Programming: Électricité de France], v. 37. Paris: Éditions Ey-
Analysis and Methods. Dover Publishing. ISBN 0- rolles 1985. xxviii+545 pp. MR 868083) French
486-43227-0. ed.). Chichester: John Wiley & Sons, Ltd. pp.
10 13 EXTERNAL LINKS

xix+650. ISBN 978-2-7430-1035-5. MR 2552933. • SIAM Journal on Optimization (SIOPT) and

(Fourth ed. Collection EDF R&D. Paris: Editions Editorial Policy
Tec & Doc 2009. xxxii+784 pp. MR 745802.
• SIAM Journal on Control and Optimization (SICON)
• Eugene Lawler (2001). Combinatorial Optimiza- and Editorial Policy
tion: Networks and Matroids. Dover. ISBN 0-486-
41453-1.

• Lawler, E. L.; Lenstra, J. K.; Rinnooy Kan, A. H.

13 External links
G.; Shmoys, D. B. (1985), The traveling salesman
problem: A guided tour of combinatorial optimiza- • COIN-OR—Computational Infrastructure for Op-
tion, John Wiley & Sons, ISBN 0-471-90413-9. erations Research

• Jon Lee; A First Course in Combinatorial Optimiza- • Decision Tree for Optimization Software Links to
tion; Cambridge University Press; 2004; ISBN 0- optimization source codes
521-01012-8. • Global optimization
• Christos H. Papadimitriou and Kenneth Steiglitz • Mathematical Programming Glossary
Combinatorial Optimization : Algorithms and Com-
plexity; Dover Pubns; (paperback, Unabridged edi- • Mathematical Programming Society
tion, July 1998) ISBN 0-486-40258-4.
• NEOS Guide currently being replaced by the NEOS
Wiki
11.4 Relaxation (extension method)
• Optimization Online A repository for optimization
Methods to obtain suitable (in some sense) natural ex- e-prints
tensions of optimization problems that otherwise lack • Optimization Related Links
of existence or stability of solutions to obtain problems
with guaranteed existence of solutions and their stabil- • Convex Optimization I EE364a: Course from Stan-
ity in some sense (typically under various perturbation of ford University
data) are in general called relaxation. Solutions of such
extended (=relaxed) problems in some sense character- • Convex Optimization – Boyd and Vandenberghe
izes (at least certain features) of the original problems, Book on Convex Optimization
e.g. as far as their optimizing sequences concerns. Re- • Book and Course on Optimization Methods for En-
laxed problems may also possesses their own natural lin- gineering Design
ear structure that may yield specific optimality conditions
different from optimality conditions for the original prob-
lems.

• H. O. Fattorini: Infinite Dimensional Optimization

and Control Theory. Cambridge Univ. Press, 1999.

• P. Pedregal: Parametrized Measures and Variational

Principles. Birkhäuser, Basel, 1997

• T. Roubicek: “Relaxation in Optimization Theory

and Variational Calculus”. W. de Gruyter, Berlin,
1997. ISBN 3-11-014542-1.

• J. Warga: Optimal control of differential and func-

tional equations. Academic Press, 1972.

12 Journals
• Computational Optimization and Applications

• Journal of Computational Optimization in Economics

and Finance

• Journal of Economic Dynamics and Control

 11

14 Text and image sources, contributors, and licenses

14.1 Text
• Mathematical optimization Source: https://en.wikipedia.org/wiki/Mathematical_optimization?oldid=726084568 Contributors: Damian
Yerrick, AxelBoldt, Zundark, The Anome, Ap, Awaterl, Peterlin~enwiki, Rade Kutil, Heron, Ryguasu, Patrick, Michael Hardy, Chan
siuman, Fred Bauder, David Martland, Karada, Stevan White, Mxn, Hike395, Jurgen~enwiki, Charles Matthews, Dysprosia, Jitse Niesen,
Topbanana, David.Monniaux, Robbot, Ojigiri~enwiki, Moink, Daniel Dickman, Giftlite, BenFrantzDale, Lethe, Bfinn, Ajgorhoe, Andris,
Jason Quinn, Wmahan, KaHa242, MarkSweep, APH, H Padleckas, Fintor, Skifreak~enwiki, PhotoBox, Mike Rosoft, Discospinster, Mat
cross, Paul August, Bender235, Saraedum, Lycurgus, Aaronbrick, Obradovic Goran, Mdd, Tsirel, Msh210, Diego Moya, Olegalexandrov,
Pontus, Ish ishwar, Artur adib, MIT Trekkie, Oleg Alexandrov, Woohookitty, Myleslong, Smmurphy, Isnow, Marudubshinki, BD2412,
JIP, Jorunn, Rjwilmsi, Tizio, Salix alba, DonSiano, Marcol, Mathbot, Nimur, Kolbasz, TeaDrinker, Alphachimp, Kri, Chobot, Roboto
de Ajvol, YurikBot, Wavelength, Deeptrivia, Encyclops, RussBot, Chaos, David R. Ingham, Voyevoda, RKUrsem, Retired username,
Hakkinen, Dsol, Gglockner, Bota47, NormDor, Arthur Rubin, MaNeMeBasat, Nojhan, RandallZ, Sardanaphalus, SmackBot, Inverse-
Hypercube, Vermorel, Mcld, Misfeldt, Oli Filth, EncMstr, Newbyman, DHN-bot~enwiki, MaxSem, Jasonb05, Ggpauly, Hua001, Brian-
boonstra, Tbbooher~enwiki, Asm4, Twocs, G.de.Lange, Antonielly, JHunterJ, Mr Stephen, Xprime, Carlo.milanesi, Hu12, Riedel~enwiki,
Polar Bear~enwiki, Lavaka, CRGreathouse, CmdrObot, Jonnat, Jackzhp, Van helsing, Thomasmeeks, Solomon Douglas, Stebulus, Schaber,
Cydebot, Mikewax, Czenek~enwiki, Headbomb, Arnab das, KrakatoaKatie, Widefox, AnAj, VictorAnyakin, JAnDbot, JPRBW, Hamster-
lopithecus, Ravelite, Roleplayer, LSpring, DRHagen, Smartcat, MartinDK, Sabamo, Mrbynum, Charlesreid1, Cic, Armehrabian, LisaTK,
David Eppstein, Martynas Patasius, Bradgib, Erkan Yilmaz, Mange01, Stochastics, Salih, Burgaz, Nwbeeson, JonMcLoone, Epistemeni-
cal, Bonadea, LastChanceToBe, VolkovBot, LokiClock, Philip Trueman, Oshwah, Anonymous Dissident, TPlantenga, Broadbot, Srinnath,
Johngcarlsson, JFPuget, Sapphic, Struway, Mangogirl2, Wikibuki, Zwgeem, Truecobb, Gerakibot, This, that and the other, Paolo.dL,
CharlesGillingham, Iknowyourider, Dattorro, Rinconsoleao, Procellarum, PipepBot, Justin W Smith, PimBeers, Suegerman, Daveagp,
Dmitrey, Auntof6, Lbertolotti, DragonBot, Dianegarey, Doobliebop, DumZiBoT, Thoughtfire, Nicoguaro, Oğuz Ergin, Sliders06, Ad-
dbot, Ablevy, Щегол, Ekojnoekoj, MrOllie, Download, HiYoSilver01, Delaszk, Kiril Simeonovski, Zorrobot, Jarble, ConstantLearner,
Luckas-bot, Ashburyjohn, Yobot, Diracula, Travis.a.buckingham, Pcap, Lylenorton, Deuxoursendormis, AnomieBOT, Erel Segal, Galou-
bet, Citation bot, ArthurBot, Pownuk, Quebec99, Obersachsebot, Xqbot, Carbaholic, P99am, Micemug, Wamanning, Isheden, Almabot,
Knillinux, Georg Stillfried, Omnipaedista, RibotBOT, Klochkov.ivan, YuriyMikhaylovskiy, Amaury, Optiy, Mcmlxxxi, Schlitz4U, Eiro06,
Ct529, Kamitsaha, FrescoBot, Nageh, X7q, Asadi1978, D'ohBot, Pschaus, Boxplot, HRoestBot, Kiefer.Wolfowitz, MaximizeMinimize,
Jmc200, FoxBot, Mdwang, Dsz4, Trappist the monk, Dpbert, Rbdevore, Kabanov Dmitry, Bpadmakumar, Duoduoduo, BlakeJRiley,
Crisgh, H.ehsaan, RjwilmsiBot, Hosseininassab, Jowa fan, EmausBot, John of Reading, Nacopt, Carbo1200, Cfg1777, Metafun, Psogeek,
Daryakav, Optimering, Krystofer, Bonnans, ZéroBot, The Nut, AManWithNoPlan, Katie O'Hare, Robbiemorrison, Edesigner, Іванко1,
Zfeinst, Syst analytic, Brycehughes, ClueBot NG, Jean-Charles.Gilbert, Chester Markel, Robiminer, Leonardo61, Frietjes, Sahinidis, Har-
rycaterpillar, Helpful Pixie Bot, Dwassel, Rxnt, BG19bot, Meldraft, Thiscomments voice, MSchlueter, FiveColourMap, Heroestr, Al-
bert triv, Osiris, Centathlete, Mdann52, ChrisGualtieri, Denmartines, Makecat-bot, Limit-theorem, Delafé, PiotrGregorczyk, SakeUPenn,
Lazars53, Buzzzman, Cubism44, Thennicke, Βελτιστοποίησης, Mehr86, Mitsisko, Tyrneaeplith, Loraof, Malcops, KasparBot, Miklomp,
Waterfall mg and Anonymous: 255

14.2 Images
• File:Commons-logo.svg Source: https://upload.wikimedia.org/wikipedia/en/4/4a/Commons-logo.svg License: CC-BY-SA-3.0 Contribu-
tors: ? Original artist: ?
• File:Folder_Hexagonal_Icon.svg Source: https://upload.wikimedia.org/wikipedia/en/4/48/Folder_Hexagonal_Icon.svg License: Cc-by-
sa-3.0 Contributors: ? Original artist: ?
• File:Max_paraboloid.svg Source: https://upload.wikimedia.org/wikipedia/commons/7/72/Max_paraboloid.svg License: CC BY-SA 4.0
Contributors: Own work Original artist: IkamusumeFan
• File:Portal-puzzle.svg Source: https://upload.wikimedia.org/wikipedia/en/f/fd/Portal-puzzle.svg License: Public domain Contributors: ?
Original artist: ?

14.3 Content license

• Creative Commons Attribution-Share Alike 3.0