PyTorch Lightning 1.1: research : CIFAR10 (VGG11, 13, 16)
作成 : (株)クラスキャット セールスインフォメーション
作成日時 : 02/19/2021 (1.1.x)
* 本ページは、以下のリソースを参考にして遂行した実験結果のレポートです：
* ご自由にリンクを張って頂いてかまいませんが、sales-info@classcat.com までご一報いただけると嬉しいです。
★ 無料セミナー実施中 ★ クラスキャット主催 人工知能 & ビジネス Web セミナー
人工知能とビジネスをテーマにウェビナー (WEB セミナー) を定期的に開催しています。スケジュールは弊社 公式 Web サイト でご確認頂けます。
- お住まいの地域に関係なく Web ブラウザからご参加頂けます。事前登録 が必要ですのでご注意ください。
- Windows PC のブラウザからご参加が可能です。スマートデバイスもご利用可能です。
クラスキャットは人工知能・テレワークに関する各種サービスを提供しております :
| 人工知能研究開発支援 | 人工知能研修サービス | テレワーク & オンライン授業を支援 |
| PoC(概念実証)を失敗させないための支援 (本支援はセミナーに参加しアンケートに回答した方を対象としています。) | ||
◆ お問合せ : 本件に関するお問い合わせ先は下記までお願いいたします。
| 株式会社クラスキャット セールス・マーケティング本部 セールス・インフォメーション |
| E-Mail：sales-info@classcat.com ; WebSite: https://www.classcat.com/ |
| Facebook: https://www.facebook.com/ClassCatJP/ |
research: CIFAR10 (VGG11, 13, 16)
結果
40 エポック : OneCycleLR
- VGG13 – {‘test_acc’: 0.8841999769210815, ‘test_loss’: 0.35507383942604065} – Wall time: 23min 5s
150 エポック: ReduceLROnPlateau
- VGG11 – {‘test_acc’: 0.9083999991416931, ‘test_loss’: 0.37741217017173767} – Wall time: 1h 6min 2s
- VGG13 – {‘test_acc’: 0.926800012588501, ‘test_loss’: 0.30548807978630066} – Wall time: 1h 23min 4s
- VGG16 – {‘test_acc’: 0.9093999862670898, ‘test_loss’: 0.37196555733680725} – Wall time: 1h 39min 4s
コード
import torch import torch.nn as nn
cfg = {
'VGG11': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
'VGG13': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
'VGG16': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],
'VGG19': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M'],
}
class VGG(nn.Module):
def __init__(self, vgg_name):
super(VGG, self).__init__()
self.features = self._make_layers(cfg[vgg_name])
self.classifier = nn.Linear(512, 10)
def forward(self, x):
out = self.features(x)
out = out.view(out.size(0), -1)
out = self.classifier(out)
return out
def _make_layers(self, cfg):
layers = []
in_channels = 3
for x in cfg:
if x == 'M':
layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
else:
layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
nn.BatchNorm2d(x),
nn.ReLU(inplace=True)]
in_channels = x
layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
return nn.Sequential(*layers)
net = VGG('VGG11')
print(net)
x = torch.randn(2,3,32,32)
y = net(x)
print(y.size())
VGG(
(features): Sequential(
(0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
(3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(4): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(5): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(6): ReLU(inplace=True)
(7): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(8): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(9): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(10): ReLU(inplace=True)
(11): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(12): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(13): ReLU(inplace=True)
(14): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(15): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(16): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(17): ReLU(inplace=True)
(18): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(19): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(20): ReLU(inplace=True)
(21): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(22): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(23): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(24): ReLU(inplace=True)
(25): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(26): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(27): ReLU(inplace=True)
(28): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(29): AvgPool2d(kernel_size=1, stride=1, padding=0)
)
(classifier): Linear(in_features=512, out_features=10, bias=True)
)
torch.Size([2, 10])
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
device
device(type='cuda')
from torchsummary import summary
summary(VGG('VGG11').to('cuda'), (3, 32, 32))
----------------------------------------------------------------
Layer (type) Output Shape Param #
================================================================
Conv2d-1 [-1, 64, 32, 32] 1,792
BatchNorm2d-2 [-1, 64, 32, 32] 128
ReLU-3 [-1, 64, 32, 32] 0
MaxPool2d-4 [-1, 64, 16, 16] 0
Conv2d-5 [-1, 128, 16, 16] 73,856
BatchNorm2d-6 [-1, 128, 16, 16] 256
ReLU-7 [-1, 128, 16, 16] 0
MaxPool2d-8 [-1, 128, 8, 8] 0
Conv2d-9 [-1, 256, 8, 8] 295,168
BatchNorm2d-10 [-1, 256, 8, 8] 512
ReLU-11 [-1, 256, 8, 8] 0
Conv2d-12 [-1, 256, 8, 8] 590,080
BatchNorm2d-13 [-1, 256, 8, 8] 512
ReLU-14 [-1, 256, 8, 8] 0
MaxPool2d-15 [-1, 256, 4, 4] 0
Conv2d-16 [-1, 512, 4, 4] 1,180,160
BatchNorm2d-17 [-1, 512, 4, 4] 1,024
ReLU-18 [-1, 512, 4, 4] 0
Conv2d-19 [-1, 512, 4, 4] 2,359,808
BatchNorm2d-20 [-1, 512, 4, 4] 1,024
ReLU-21 [-1, 512, 4, 4] 0
MaxPool2d-22 [-1, 512, 2, 2] 0
Conv2d-23 [-1, 512, 2, 2] 2,359,808
BatchNorm2d-24 [-1, 512, 2, 2] 1,024
ReLU-25 [-1, 512, 2, 2] 0
Conv2d-26 [-1, 512, 2, 2] 2,359,808
BatchNorm2d-27 [-1, 512, 2, 2] 1,024
ReLU-28 [-1, 512, 2, 2] 0
MaxPool2d-29 [-1, 512, 1, 1] 0
AvgPool2d-30 [-1, 512, 1, 1] 0
Linear-31 [-1, 10] 5,130
================================================================
Total params: 9,231,114
Trainable params: 9,231,114
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.01
Forward/backward pass size (MB): 3.71
Params size (MB): 35.21
Estimated Total Size (MB): 38.94
----------------------------------------------------------------
OneCycleLR スケジューラ
! pip install pytorch-lightning pytorch-lightning-bolts -qU
import torch import torch.nn as nn import torch.nn.functional as F from torch.optim.lr_scheduler import OneCycleLR from torch.optim.swa_utils import AveragedModel, update_bn import torchvision import pytorch_lightning as pl from pytorch_lightning.callbacks import LearningRateMonitor from pytorch_lightning.metrics.functional import accuracy from pl_bolts.datamodules import CIFAR10DataModule from pl_bolts.transforms.dataset_normalizations import cifar10_normalization
pl.seed_everything(7);
Global seed set to 7
batch_size = 32
train_transforms = torchvision.transforms.Compose([
torchvision.transforms.RandomCrop(32, padding=4),
torchvision.transforms.RandomHorizontalFlip(),
torchvision.transforms.ToTensor(),
cifar10_normalization(),
])
test_transforms = torchvision.transforms.Compose([
torchvision.transforms.ToTensor(),
cifar10_normalization(),
])
cifar10_dm = CIFAR10DataModule(
batch_size=batch_size,
train_transforms=train_transforms,
test_transforms=test_transforms,
val_transforms=test_transforms,
)
class LitCifar10VGG(pl.LightningModule):
def __init__(self, model_name='vgg11', lr=0.05):
super().__init__()
self.save_hyperparameters()
self.model = VGG(model_name)
def forward(self, x):
out = self.model(x)
return F.log_softmax(out, dim=1)
def training_step(self, batch, batch_idx):
x, y = batch
logits = F.log_softmax(self.model(x), dim=1)
loss = F.nll_loss(logits, y)
self.log('train_loss', loss)
return loss
def evaluate(self, batch, stage=None):
x, y = batch
logits = self(x)
loss = F.nll_loss(logits, y)
preds = torch.argmax(logits, dim=1)
acc = accuracy(preds, y)
if stage:
self.log(f'{stage}_loss', loss, prog_bar=True)
self.log(f'{stage}_acc', acc, prog_bar=True)
def validation_step(self, batch, batch_idx):
self.evaluate(batch, 'val')
def test_step(self, batch, batch_idx):
self.evaluate(batch, 'test')
def configure_optimizers(self):
optimizer = torch.optim.SGD(self.parameters(), lr=self.hparams.lr, momentum=0.9, weight_decay=5e-4)
steps_per_epoch = 45000 // batch_size
scheduler_dict = {
'scheduler': OneCycleLR(optimizer, 0.1, epochs=self.trainer.max_epochs, steps_per_epoch=steps_per_epoch),
'interval': 'step',
}
return {'optimizer': optimizer, 'lr_scheduler': scheduler_dict}
%%time
model = LitCifar10VGG(model_name='VGG13', lr=0.05)
model.datamodule = cifar10_dm
trainer = pl.Trainer(
gpus=1,
max_epochs=40,
auto_scale_batch_size=True,
auto_lr_find = True,
progress_bar_refresh_rate=20,
logger=pl.loggers.TensorBoardLogger('lightning_logs/', name='vgg13'),
callbacks=[LearningRateMonitor(logging_interval='step')],
)
trainer.fit(model, cifar10_dm)
trainer.test(model, datamodule=cifar10_dm);
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
| Name | Type | Params
-------------------------------
0 | model | VGG | 9.4 M
-------------------------------
9.4 M Trainable params
0 Non-trainable params
9.4 M Total params
37.664 Total estimated model params size (MB)
...
--------------------------------------------------------------------------------
DATALOADER:0 TEST RESULTS
{'test_acc': 0.8841999769210815, 'test_loss': 0.35507383942604065}
--------------------------------------------------------------------------------
CPU times: user 17min 9s, sys: 2min 47s, total: 19min 57s
Wall time: 23min 5s
# Start tensorboard. %reload_ext tensorboard %tensorboard --logdir lightning_logs/
ReduceLROnPlateau スケジューラ
import torch import torch.nn as nn import torch.nn.functional as F from torch.optim.lr_scheduler import OneCycleLR, CyclicLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau from torch.optim.swa_utils import AveragedModel, update_bn import torchvision import pytorch_lightning as pl from pytorch_lightning.callbacks import LearningRateMonitor, GPUStatsMonitor, EarlyStopping from pytorch_lightning.metrics.functional import accuracy from pl_bolts.datamodules import CIFAR10DataModule from pl_bolts.transforms.dataset_normalizations import cifar10_normalization
pl.seed_everything(7);
batch_size = 50
train_transforms = torchvision.transforms.Compose([
torchvision.transforms.RandomCrop(32, padding=4),
torchvision.transforms.RandomHorizontalFlip(),
torchvision.transforms.ToTensor(),
cifar10_normalization(),
])
test_transforms = torchvision.transforms.Compose([
torchvision.transforms.ToTensor(),
cifar10_normalization(),
])
cifar10_dm = CIFAR10DataModule(
batch_size=batch_size,
train_transforms=train_transforms,
test_transforms=test_transforms,
val_transforms=test_transforms,
)
class LitCifar10(pl.LightningModule):
def __init__(self, model_name='vgg11', lr=0.05):
super().__init__()
self.save_hyperparameters()
self.model = VGG(model_name)
def forward(self, x):
out = self.model(x)
return F.log_softmax(out, dim=1)
def training_step(self, batch, batch_idx):
x, y = batch
logits = F.log_softmax(self.model(x), dim=1)
loss = F.nll_loss(logits, y)
self.log('train_loss', loss)
return loss
def evaluate(self, batch, stage=None):
x, y = batch
logits = self(x)
loss = F.nll_loss(logits, y)
preds = torch.argmax(logits, dim=1)
acc = accuracy(preds, y)
if stage:
self.log(f'{stage}_loss', loss, prog_bar=True)
self.log(f'{stage}_acc', acc, prog_bar=True)
def validation_step(self, batch, batch_idx):
self.evaluate(batch, 'val')
def test_step(self, batch, batch_idx):
self.evaluate(batch, 'test')
def configure_optimizers(self):
if False:
optimizer = torch.optim.Adam(self.parameters(), lr=self.hparams.lr, weight_decay=0, eps=1e-3)
else:
optimizer = torch.optim.SGD(self.parameters(), lr=self.hparams.lr, momentum=0.9, weight_decay=5e-4)
return {
'optimizer': optimizer,
'lr_scheduler': ReduceLROnPlateau(optimizer, 'max', patience=4, factor=0.8, verbose=True, threshold=0.0001, threshold_mode='abs', cooldown=1, min_lr=1e-5),
'monitor': 'val_acc'
}
def xconfigure_optimizers(self):
#print("###")
#print(self.hparams)
optimizer = torch.optim.SGD(self.parameters(), lr=self.hparams.lr, momentum=0.9, weight_decay=5e-4)
steps_per_epoch = 45000 // batch_size
scheduler_dict = {
#'scheduler': ExponentialLR(optimizer, gamma=0.1),
#'interval': 'epoch',
'scheduler': OneCycleLR(optimizer, max_lr=0.1, pct_start=0.2, epochs=self.trainer.max_epochs, steps_per_epoch=steps_per_epoch),
#'scheduler': CyclicLR(optimizer, base_lr=0.001, max_lr=0.1, step_size_up=steps_per_epoch*2, mode="triangular2"),
#'scheduler': CyclicLR(optimizer, base_lr=0.001, max_lr=0.1, step_size_up=steps_per_epoch, mode="exp_range", gamma=0.85),
#'scheduler': CosineAnnealingLR(optimizer, T_max=200),
'interval': 'step',
}
return {'optimizer': optimizer, 'lr_scheduler': scheduler_dict}
VGG11
%%time
model = LitCifar10(model_name='VGG11', lr=0.05)
model.datamodule = cifar10_dm
trainer = pl.Trainer(
gpus=1,
max_epochs=150,
auto_scale_batch_size=True,
auto_lr_find = True,
progress_bar_refresh_rate=50,
logger=pl.loggers.TensorBoardLogger('tblogs/', name='vgg11'),
callbacks=[LearningRateMonitor(logging_interval='step')],
)
trainer.fit(model, cifar10_dm)
trainer.test(model, datamodule=cifar10_dm);
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
Files already downloaded and verified
Files already downloaded and verified
| Name | Type | Params
-------------------------------
0 | model | VGG | 9.2 M
-------------------------------
9.2 M Trainable params
0 Non-trainable params
9.2 M Total params
36.924 Total estimated model params size (MB)
(...)
Epoch 26: reducing learning rate of group 0 to 4.0000e-02.
Epoch 36: reducing learning rate of group 0 to 3.2000e-02.
Epoch 45: reducing learning rate of group 0 to 2.5600e-02.
Epoch 53: reducing learning rate of group 0 to 2.0480e-02.
Epoch 59: reducing learning rate of group 0 to 1.6384e-02.
Epoch 65: reducing learning rate of group 0 to 1.3107e-02.
Epoch 76: reducing learning rate of group 0 to 1.0486e-02.
Epoch 82: reducing learning rate of group 0 to 8.3886e-03.
Epoch 89: reducing learning rate of group 0 to 6.7109e-03.
Epoch 95: reducing learning rate of group 0 to 5.3687e-03.
Epoch 101: reducing learning rate of group 0 to 4.2950e-03.
Epoch 110: reducing learning rate of group 0 to 3.4360e-03.
Epoch 118: reducing learning rate of group 0 to 2.7488e-03.
Epoch 126: reducing learning rate of group 0 to 2.1990e-03.
Epoch 134: reducing learning rate of group 0 to 1.7592e-03.
Epoch 142: reducing learning rate of group 0 to 1.4074e-03.
Epoch 148: reducing learning rate of group 0 to 1.1259e-03.
(...)
--------------------------------------------------------------------------------
DATALOADER:0 TEST RESULTS
{'test_acc': 0.9083999991416931, 'test_loss': 0.37741217017173767}
--------------------------------------------------------------------------------
CPU times: user 39min 52s, sys: 11min 7s, total: 51min
Wall time: 1h 6min 2s
VGG13
%%time
model = LitCifar10(model_name='VGG13', lr=0.05)
model.datamodule = cifar10_dm
trainer = pl.Trainer(
gpus=1,
max_epochs=150,
auto_scale_batch_size=True,
auto_lr_find = True,
progress_bar_refresh_rate=50,
logger=pl.loggers.TensorBoardLogger('tblogs/', name='vgg13'),
callbacks=[LearningRateMonitor(logging_interval='step')],
)
trainer.fit(model, cifar10_dm)
trainer.test(model, datamodule=cifar10_dm);
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
| Name | Type | Params
-------------------------------
0 | model | VGG | 9.4 M
-------------------------------
9.4 M Trainable params
0 Non-trainable params
9.4 M Total params
37.664 Total estimated model params size (MB)
(...)
Epoch 24: reducing learning rate of group 0 to 4.0000e-02.
Epoch 35: reducing learning rate of group 0 to 3.2000e-02.
Epoch 42: reducing learning rate of group 0 to 2.5600e-02.
Epoch 53: reducing learning rate of group 0 to 2.0480e-02.
Epoch 61: reducing learning rate of group 0 to 1.6384e-02.
Epoch 68: reducing learning rate of group 0 to 1.3107e-02.
Epoch 74: reducing learning rate of group 0 to 1.0486e-02.
Epoch 80: reducing learning rate of group 0 to 8.3886e-03.
Epoch 88: reducing learning rate of group 0 to 6.7109e-03.
Epoch 94: reducing learning rate of group 0 to 5.3687e-03.
Epoch 103: reducing learning rate of group 0 to 4.2950e-03.
Epoch 109: reducing learning rate of group 0 to 3.4360e-03.
Epoch 115: reducing learning rate of group 0 to 2.7488e-03.
Epoch 126: reducing learning rate of group 0 to 2.1990e-03.
Epoch 137: reducing learning rate of group 0 to 1.7592e-03.
Epoch 148: reducing learning rate of group 0 to 1.4074e-03.
(...)
--------------------------------------------------------------------------------
DATALOADER:0 TEST RESULTS
{'test_acc': 0.926800012588501, 'test_loss': 0.30548807978630066}
--------------------------------------------------------------------------------
CPU times: user 54min 36s, sys: 20min 29s, total: 1h 15min 6s
Wall time: 1h 23min 4s
VGG16
%%time
model = LitCifar10(model_name='VGG16', lr=0.05)
model.datamodule = cifar10_dm
trainer = pl.Trainer(
gpus=1,
max_epochs=150,
auto_scale_batch_size=True,
auto_lr_find = True,
progress_bar_refresh_rate=50,
logger=pl.loggers.TensorBoardLogger('tblogs/', name='vgg16'),
callbacks=[LearningRateMonitor(logging_interval='step')],
)
trainer.fit(model, cifar10_dm)
trainer.test(model, datamodule=cifar10_dm);
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
| Name | Type | Params
-------------------------------
0 | model | VGG | 14.7 M
-------------------------------
14.7 M Trainable params
0 Non-trainable params
14.7 M Total params
58.913 Total estimated model params size (MB)
(...)
Epoch 24: reducing learning rate of group 0 to 4.0000e-02.
Epoch 41: reducing learning rate of group 0 to 3.2000e-02.
Epoch 47: reducing learning rate of group 0 to 2.5600e-02.
Epoch 64: reducing learning rate of group 0 to 2.0480e-02.
Epoch 70: reducing learning rate of group 0 to 1.6384e-02.
Epoch 87: reducing learning rate of group 0 to 1.3107e-02.
Epoch 93: reducing learning rate of group 0 to 1.0486e-02.
Epoch 100: reducing learning rate of group 0 to 8.3886e-03.
Epoch 109: reducing learning rate of group 0 to 6.7109e-03.
Epoch 120: reducing learning rate of group 0 to 5.3687e-03.
Epoch 129: reducing learning rate of group 0 to 4.2950e-03.
Epoch 138: reducing learning rate of group 0 to 3.4360e-03.
Epoch 150: reducing learning rate of group 0 to 2.7488e-03.
(...)
--------------------------------------------------------------------------------
DATALOADER:0 TEST RESULTS
{'test_acc': 0.9093999862670898, 'test_loss': 0.37196555733680725}
--------------------------------------------------------------------------------
CPU times: user 1h 7min 4s, sys: 24min 9s, total: 1h 31min 13s
Wall time: 1h 39min 4s
以上