Subgradient Method
Ryan Tibshirani
Convex Optimization 10-725
Last last time: gradient descent
Consider the problem
min f (x)
x
for f convex and differentiable, dom(f ) = Rn . Gradient descent:
choose initial x(0) ∈ Rn , repeat:
x(k) = x(k−1) − tk · ∇f (x(k−1) ), k = 1, 2, 3, . . .
Step sizes tk chosen to be fixed and small, or by backtracking line
search
If ∇f is Lipschitz, gradient descent has convergence rate O(1/).
Downsides:
• Requires f differentiable — addressed this lecture
• Can be slow to converge — addressed next lecture
2
Subgradient method
Now consider f convex, having dom(f ) = Rn , but not necessarily
differentiable
Subgradient method: like gradient descent, but replacing gradients
with subgradients. Initialize x(0) , repeat:
x(k) = x(k−1) − tk · g (k−1) , k = 1, 2, 3, . . .
where g (k−1) ∈ ∂f (x(k−1) ), any subgradient of f at x(k−1)
Subgradient method is not necessarily a descent method, thus we
(k)
keep track of best iterate xbest among x(0) , . . . , x(k) so far, i.e.,
(k)
f (xbest ) = min f (x(i) )
i=0,...,k
3
Outline
Today:
• How to choose step sizes
• Convergence analysis
• Intersection of sets
• Projected subgradient method
4
Step size choices
• Fixed step sizes: tk = t all k = 1, 2, 3, . . .
• Diminishing step sizes: choose to meet conditions
∞
X ∞
X
t2k < ∞, tk = ∞,
k=1 k=1
i.e., square summable but not summable. Important here that
step sizes go to zero, but not too fast
There are several other options too, but key difference to gradient
descent: step sizes are pre-specified, not adaptively computed
5
Convergence analysis
Assume that f convex, dom(f ) = Rn , and also that f is Lipschitz
continuous with constant G > 0, i.e.,
|f (x) − f (y)| ≤ Gkx − yk2 for all x, y
Theorem: For a fixed step size t, subgradient method satisfies
(k)
lim f (xbest ) ≤ f ? + G2 t/2
k→∞
Theorem: For diminishing step sizes, subgradient method sat-
isfies
(k)
lim f (xbest ) = f ?
k→∞
6
Basic inequality
Can prove both results from same basic inequality. Key steps:
• Using definition of subgradient,
kx(k) − x? k22 ≤
kx(k−1) − x? k22 − 2tk f (x(k−1) ) − f (x? ) + t2k kg (k−1) k22
• Iterating last inequality,
kx(k) − x? k22 ≤
k
X k
X
kx(0) − x? k22 − 2 ti f (x(i−1) ) − f (x? ) + t2i kg (i−1) k22
i=1 i=1
7
• Using kx(k) − x? k2 ≥ 0, and letting R = kx(0) − x? k2 ,
k
X k
X
0 ≤ R2 − 2 ti f (x(i−1) ) − f (x? ) + G2 t2i
i=1 i=1
(k)
• Introducing f (xbest ) = mini=0,...,k f (x(i) ), and rearranging, we
have the basic inequality
R2 + G2 ki=1 t2i
P
(k) ?
f (xbest ) − f (x ) ≤
2 ki=1 ti
P
For different step sizes choices, convergence results can be directly
obtained from this bound, e.g., previous theorems follow
8
Convergence rate
The basic inequality tells us that after k steps, we have
R2 + G2 ki=1 t2i
P
(k) ?
f (xbest ) − f (x ) ≤
2 ki=1 ti
P
With fixed step size t, this gives
(k) R2 G2 t
f (xbest ) − f ? ≤ +
2kt 2
For this to be ≤ , let’s make each term ≤ /2. So we can choose
t = /G2 , and k = R2 /t · 1/ = R2 G2 /2
That is, subgradient method has convergence rate O(1/2 ) ... note
that this is slower than O(1/) rate of gradient descent
9
Example: regularized logistic regression
Given (xi , yi ) ∈ Rp × {0, 1} for i = 1, . . . , n, the logistic regression
loss is
Xn
f (β) = − yi xTi β + log(1 + exp(xTi β))
i=1
This is a smooth and convex function with
n
X
∇f (β) = yi − pi (β) xi
i=1
where pi (β) = exp(xTi β)/(1 + exp(xTi β)), i = 1, . . . , n. Consider
the regularized problem:
min f (β) + λ · P (β)
β
where P (β) = kβk22 , ridge penalty; or P (β) = kβk1 , lasso penalty
10
Ridge: use gradients; lasso: use subgradients. Example here has
n = 1000, p = 20:
Gradient descent Subgradient method
t=0.001 t=0.001
2.00
1e−01
t=0.001/k
0.50
1e−04
f−fstar
f−fstar
0.20
1e−07
1e−10
0.05
0.02
1e−13
0 50 100 150 200 0 50 100 150 200
k k
Step sizes hand-tuned to be favorable for each method (of course
comparison is imperfect, but it reveals the convergence behaviors)
11
Polyak step sizes
Polyak step sizes: when the optimal value f ? is known, take
f (x(k−1) ) − f ?
tk = , k = 1, 2, 3, . . .
kg (k−1) k22
Can be motivated from first step in subgradient proof:
kx(k) −x? k22 ≤ kx(k−1) −x? k22 −2tk f (x(k−1) )−f (x? ) +t2k kg (k−1) k22
Polyak step size minimizes the right-hand side
With Polyak step sizes, can show subgradient method converges to
optimal value. Convergence rate is still O(1/2 )
12
Example: intersection of sets
Suppose we want to find x? ∈ C1 ∩ · · · ∩ Cm , i.e., find a point in
intersection of closed, convex sets C1 , . . . , Cm
First define
fi (x) = dist(x, Ci ), i = 1, . . . , m
f (x) = max fi (x)
i=1,...,m
and now solve
min f (x)
x
Check: is this convex?
Note that f ? = 0 ⇐⇒ x? ∈ C1 ∩ · · · ∩ Cm
13
Recall the distance function dist(x, C) = miny∈C ky − xk2 . Last
time we computed its gradient
x − PC (x)
∇dist(x, C) =
kx − PC (x)k2
where PC (x) is the projection of x onto C
Also recall subgradient rule: if f (x) = maxi=1,...,m fi (x), then
[
∂f (x) = conv ∂fi (x)
i:fi (x)=f (x)
So if fi (x) = f (x) and gi ∈ ∂fi (x), then gi ∈ ∂f (x)
14
Put these two facts together for intersection of sets problem, with
fi (x) = dist(x, Ci ): if Ci is farthest set from x (so fi (x) = f (x)),
and
x − PCi (x)
gi = ∇fi (x) =
kx − PCi (x)k2
then gi ∈ ∂f (x)
Now apply subgradient method, with Polyak size tk = f (x(k−1) ).
At iteration k, with Ci farthest from x(k−1) , we perform update
x(k−1) − PCi (x(k−1) )
x(k) = x(k−1) − f (x(k−1) )
kx(k−1) − PCi (x(k−1) )k2
= PCi (x(k−1) )
15
For two sets, this is the famous alternating projections algorithm1 ,
i.e., just keep projecting back and forth
(From Boyd’s lecture notes)
1
von Neumann (1950), “Functional operators, volume II: The geometry of
orthogonal spaces”
16
Projected subgradient method
To optimize a convex function f over a convex set C,
min f (x) subject to x ∈ C
x
we can use the projected subgradient method. Just like the usual
subgradient method, except we project onto C at each iteration:
x(k) = PC x(k−1) − tk · g (k−1) , k = 1, 2, 3, . . .
Assuming we can do this projection, we get the same convergence
guarantees as the usual subgradient method, with the same step
size choices
17
What sets C are easy to project onto? Lots, e.g.,
• Affine images: {Ax + b : x ∈ Rn }
• Solution set of linear system: {x : Ax = b}
• Nonnegative orthant: Rn+ = {x : x ≥ 0}
• Some norm balls: {x : kxkp ≤ 1} for p = 1, 2, ∞
• Some simple polyhedra and simple cones
Warning: it is easy to write down seemingly simple set C, and PC
can turn out to be very hard! E.g., generally hard to project onto
arbitrary polyhedron C = {x : Ax ≤ b}
Note: projected gradient descent works too, more next time ...
18
Can we do better?
Upside of the subgradient method: broad applicability. Downside:
O(1/2 ) convergence rate over problem class of convex, Lipschitz
functions is really slow
Nonsmooth first-order methods: iterative methods updating x(k) in
x(0) + span{g (0) , g (1) , . . . , g (k−1) }
where subgradients g (0) , g (1) , . . . , g (k−1) come from weak oracle
Theorem (Nesterov): For any k ≤ n−1 and starting point x(0) ,
there is a function in the problem class such that any nonsmooth
first-order method satisfies
RG
f (x(k) ) − f ? ≥ √
2(1 + k + 1)
19
Improving on the subgradient method
In words, we cannot do better than the O(1/2 ) rate of subgradient
method (unless we go beyond nonsmooth first-order methods)
So instead of trying to improve across the board, we will focus on
minimizing composite functions of the form
f (x) = g(x) + h(x)
where g is convex and differentiable, h is convex and nonsmooth
but “simple”
For a lot of problems (i.e., functions h), we can recover the O(1/)
rate of gradient descent with a simple algorithm, having important
practical consequences
20
References and further reading
• S. Boyd, Lecture notes for EE 264B, Stanford University,
Spring 2010-2011
• Y. Nesterov (1998), “Introductory lectures on convex
optimization: a basic course”, Chapter 3
• B. Polyak (1987), “Introduction to optimization”, Chapter 5
• L. Vandenberghe, Lecture notes for EE 236C, UCLA, Spring
2011-2012
21
