LASSO坐标下降法Coordinate Descent Method公式推导及代码

LASSO by Coordinate Descent Method

Prepare:

from itertools import cycle
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import lasso_path, enet_path
from sklearn import datasets
from copy import deepcopy

X = np.random.randn(100,10)
y = np.dot(X,[1,2,3,4,5,6,7,8,9,10])

The code is the simplified version of _cd_fast.enet_coordinate_descent() with beta=0 and l1_ratio=1 from scikit-learn (source code:lasso coodinate descent source). The original code is implemented in Cython, and code here is pure python for convenience, easier to understand but much slower.

Coordinate Descent Method Framework

1. randomly set ${\beta }^{(0)}$ for iteration 0
2. For $k$th iteration:
   ----For $j=1$ to $p$:
   --------${\beta }_j^{(k)}=argmi{n}_{{\beta }_j}{\mathcal{L}}_{l1}(\beta )=argmi{n}_{{\beta }_j}{\mathcal{L}}_{l1}({\beta }_1^{(k-1)},{\beta }_2^{(k-1)},\dots ,{\beta }_{j-1}^{(k-1)},{\beta }_j,{\beta }_{j+1}^{(k)},\dots ,{\beta }_p^{(k-1)})$
   ----Endfor
   ----Check convergence: if yes, end algorithm; else continue update
   Endfor

Here the objective function is
$${\mathcal{L}}_{l1}=\frac{1}{2N}(Y-X\beta {)}^T(Y-X\beta )+\lambda {\Vert \beta \Vert }_1$$
where the size of $X$, $Y$, $\beta$ is $N\times p$, $N\times 1$, $p\times 1$, which means $N$ samples and $p$ features.

Coordinate Descent Method Update Details

To update the ${\beta }_j$, we need to find the ${\beta }_j^{\ast }\text{ s.t.}\frac{\partial {\mathcal{L}}_{l1}(\beta )}{\partial {\beta }_j}=0$.

Given