{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 小批量随机梯度下降\n",
"\n",
"在每一次迭代中,梯度下降使用整个训练数据集来计算梯度,因此它有时也被称为批量梯度下降(batch gradient descent)。而随机梯度下降在每次迭代中只随机采样一个样本来计算梯度。正如我们在前几章中所看到的,我们还可以在每轮迭代中随机均匀采样多个样本来组成一个小批量,然后使用这个小批量来计算梯度。下面就来描述小批量随机梯度下降。\n",
"\n",
"\n",
"设目标函数$f(\\boldsymbol{x}): \\mathbb{R}^d \\rightarrow \\mathbb{R}$。在迭代开始前的时间步设为0。该时间步的自变量记为$\\boldsymbol{x}_0\\in \\mathbb{R}^d$,通常由随机初始化得到。在接下来的每一个时间步$t>0$中,小批量随机梯度下降随机均匀采样一个由训练数据样本索引组成的小批量$\\mathcal{B}_t$。我们可以通过重复采样(sampling with replacement)或者不重复采样(sampling without replacement)得到一个小批量中的各个样本。前者允许同一个小批量中出现重复的样本,后者则不允许如此,且更常见。对于这两者间的任一种方式,都可以使用\n",
"\n",
"$$\\boldsymbol{g}_t \\leftarrow \\nabla f_{\\mathcal{B}_t}(\\boldsymbol{x}_{t-1}) = \\frac{1}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}_t}\\nabla f_i(\\boldsymbol{x}_{t-1})$$\n",
"\n",
"来计算时间步$t$的小批量$\\mathcal{B}_t$上目标函数位于$\\boldsymbol{x}_{t-1}$处的梯度$\\boldsymbol{g}_t$。这里$|\\mathcal{B}|$代表批量大小,即小批量中样本的个数,是一个超参数。同随机梯度一样,重复采样所得的小批量随机梯度$\\boldsymbol{g}_t$也是对梯度$\\nabla f(\\boldsymbol{x}_{t-1})$的无偏估计。给定学习率$\\eta_t$(取正数),小批量随机梯度下降对自变量的迭代如下:\n",
"\n",
"$$\\boldsymbol{x}_t \\leftarrow \\boldsymbol{x}_{t-1} - \\eta_t \\boldsymbol{g}_t.$$\n",
"\n",
"基于随机采样得到的梯度的方差在迭代过程中无法减小,因此在实际中,(小批量)随机梯度下降的学习率可以在迭代过程中自我衰减,例如$\\eta_t=\\eta t^\\alpha$(通常$\\alpha=-1$或者$-0.5$)、$\\eta_t = \\eta \\alpha^t$(如$\\alpha=0.95$)或者每迭代若干次后将学习率衰减一次。如此一来,学习率和(小批量)随机梯度乘积的方差会减小。而梯度下降在迭代过程中一直使用目标函数的真实梯度,无须自我衰减学习率。\n",
"\n",
"\n",
"小批量随机梯度下降中每次迭代的计算开销为$\\mathcal{O}(|\\mathcal{B}|)$。当批量大小为1时,该算法即随机梯度下降;当批量大小等于训练数据样本数时,该算法即梯度下降。当批量较小时,每次迭代中使用的样本少,这会导致并行处理和内存使用效率变低。这使得在计算同样数目样本的情况下比使用更大批量时所花时间更多。当批量较大时,每个小批量梯度里可能含有更多的冗余信息。为了得到较好的解,批量较大时比批量较小时需要计算的样本数目可能更多,例如增大迭代周期数。\n",
"\n",
"\n",
"## 读取数据集\n",
"\n",
"本章里我们将使用一个来自NASA的测试不同飞机机翼噪音的数据集来比较各个优化算法 [1]。我们使用该数据集的前1,500个样本和5个特征,并使用标准化对数据进行预处理。"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"attributes": {
"classes": [],
"id": "",
"n": "1"
}
},
"outputs": [
{
"data": {
"text/plain": [
"(1500, 5)"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%matplotlib inline\n",
"import d2lzh as d2l\n",
"from mxnet import autograd, gluon, init, nd\n",
"from mxnet.gluon import nn, data as gdata, loss as gloss\n",
"import numpy as np\n",
"import time\n",
"\n",
"def get_data_ch7(): # 本函数已保存在d2lzh包中方便以后使用\n",
" data = np.genfromtxt('../data/airfoil_self_noise.dat', delimiter='\\t')\n",
" data = (data - data.mean(axis=0)) / data.std(axis=0)\n",
" return nd.array(data[:1500, :-1]), nd.array(data[:1500, -1])\n",
"\n",
"features, labels = get_data_ch7()\n",
"features.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 从零开始实现\n",
"\n",
"[“线性回归的从零开始实现”](../chapter_deep-learning-basics/linear-regression-scratch.ipynb)一节中已经实现过小批量随机梯度下降算法。我们在这里将它的输入参数变得更加通用,主要是为了方便本章后面介绍的其他优化算法也可以使用同样的输入。具体来说,我们添加了一个状态输入`states`并将超参数放在字典`hyperparams`里。此外,我们将在训练函数里对各个小批量样本的损失求平均,因此优化算法里的梯度不需要除以批量大小。"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"attributes": {
"classes": [],
"id": "",
"n": "3"
}
},
"outputs": [],
"source": [
"def sgd(params, states, hyperparams):\n",
" for p in params:\n",
" p[:] -= hyperparams['lr'] * p.grad"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"下面实现一个通用的训练函数,以方便本章后面介绍的其他优化算法使用。它初始化一个线性回归模型,然后可以使用小批量随机梯度下降以及后续小节介绍的其他算法来训练模型。"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"attributes": {
"classes": [],
"id": "",
"n": "4"
}
},
"outputs": [],
"source": [
"# 本函数已保存在d2lzh包中方便以后使用\n",
"def train_ch7(trainer_fn, states, hyperparams, features, labels,\n",
" batch_size=10, num_epochs=2):\n",
" # 初始化模型\n",
" net, loss = d2l.linreg, d2l.squared_loss\n",
" w = nd.random.normal(scale=0.01, shape=(features.shape[1], 1))\n",
" b = nd.zeros(1)\n",
" w.attach_grad()\n",
" b.attach_grad()\n",
"\n",
" def eval_loss():\n",
" return loss(net(features, w, b), labels).mean().asscalar()\n",
"\n",
" ls = [eval_loss()]\n",
" data_iter = gdata.DataLoader(\n",
" gdata.ArrayDataset(features, labels), batch_size, shuffle=True)\n",
" for _ in range(num_epochs):\n",
" start = time.time()\n",
" for batch_i, (X, y) in enumerate(data_iter):\n",
" with autograd.record():\n",
" l = loss(net(X, w, b), y).mean() # 使用平均损失\n",
" l.backward()\n",
" trainer_fn([w, b], states, hyperparams) # 迭代模型参数\n",
" if (batch_i + 1) * batch_size % 100 == 0:\n",
" ls.append(eval_loss()) # 每100个样本记录下当前训练误差\n",
" # 打印结果和作图\n",
" print('loss: %f, %f sec per epoch' % (ls[-1], time.time() - start))\n",
" d2l.set_figsize()\n",
" d2l.plt.plot(np.linspace(0, num_epochs, len(ls)), ls)\n",
" d2l.plt.xlabel('epoch')\n",
" d2l.plt.ylabel('loss')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"当批量大小为样本总数1,500时,优化使用的是梯度下降。梯度下降的1个迭代周期对模型参数只迭代1次。可以看到6次迭代后目标函数值(训练损失)的下降趋向了平稳。"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"attributes": {
"classes": [],
"id": "",
"n": "5"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"loss: 0.246095, 0.027039 sec per epoch\n"
]
},
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n",
"\n"
],
"text/plain": [
"