scikit-learn是Python中最好的機器學習函式庫，而PyTorch又為我們建構模型提供了方便的操作，能否將它們的優點整合起來呢？在本文中，我們將介紹如何使用scikit-learn中的網格搜尋功能來調整PyTorch 深度學習模型的超參數:
#要讓PyTorch 模型可以在scikit-learn 中使用的一個最簡單的方法是使用skorch套件。這個套件為 PyTorch 模型提供與 scikit-learn 相容的 API。在skorch中，有分類神經網路的NeuralNetClassifier和回歸神經網路的NeuralNetRegressor。
pip install skorch
要使用這些包裝器，必須使用 nn.Module 將 PyTorch 模型定義為類，然後在建構 NeuralNetClassifier 類別時將類別的名稱傳遞給模組參數。例如：
class MyClassifier(nn.Module): def __init__(self): super().__init__() ... def forward(self, x): ... return x # create the skorch wrapper model = NeuralNetClassifier( module=MyClassifier )
NeuralNetClassifier 類別的建構子可以獲得傳遞給 model.fit() 呼叫的參數（在 scikit-learn 模型中呼叫訓練循環的方法），例如輪次數和批次大小等。例如：
model = NeuralNetClassifier( module=MyClassifier, max_epochs=150, batch_size=10 )
NeuralNetClassifier類別的建構子也可以接受新的參數，這些參數可以傳遞給你的模型類別的建構函數，要求是必須在它前面加上module__(兩個下劃線)。這些新參數可能在建構函數中帶有預設值，但當包裝器實例化模型時，它們將被覆寫。例如:
import torch.nn as nn from skorch import NeuralNetClassifier class SonarClassifier(nn.Module): def __init__(self, n_layers=3): super().__init__() self.layers = [] self.acts = [] for i in range(n_layers): self.layers.append(nn.Linear(60, 60)) self.acts.append(nn.ReLU()) self.add_module(f"layer{i}", self.layers[-1]) self.add_module(f"act{i}", self.acts[-1]) self.output = nn.Linear(60, 1) def forward(self, x): for layer, act in zip(self.layers, self.acts): x = act(layer(x)) x = self.output(x) return x model = NeuralNetClassifier( module=SonarClassifier, max_epochs=150, batch_size=10, module__n_layers=2 )
我們可以透過初始化一個模型並列印來驗證結果:
print(model.initialize()) #结果如下：[initialized]( module_=SonarClassifier( (layer0): Linear(in_features=60, out_features=60, bias=True) (act0): ReLU() (layer1): Linear(in_features=60, out_features=60, bias=True) (act1): ReLU() (output): Linear(in_features=60, out_features=1, bias=True) ), )
網格搜尋是一種模型超參數優化技術。它只是簡單地窮盡超參數的所有組合，並找到給出最佳分數的組合。在scikit-learn中，GridSearchCV類別提供了這種技術。在建構這個類別時，必須在param_grid參數中提供一個超參數字典。這是模型參數名稱和要嘗試的值數組的映射。
預設使用精確度作為最佳化的分數，但其他分數可以在GridSearchCV建構子的score參數中指定。 GridSearchCV將為每個參數組合建立一個模型進行評估。並且使用預設的3倍交叉驗證，這些都是可以透過參數來進行設定的。
下面是定義一個簡單網格搜尋的範例:
param_grid = { 'epochs': [10,20,30] } grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3) grid_result = grid.fit(X, Y)
透過將GridSearchCV建構函式中的n_jobs參數設定為 -1表示將使用機器上的所有核心。否則，網格搜尋進程將只在單執行緒中運行，這在多核心cpu中較慢。
運行完畢就可以在grid.fit()傳回的結果物件中存取網格搜尋的結果。 best_score提供了在最佳化過程中觀察到的最佳分數，best_params_描述了獲得最佳結果的參數組合。
我們的範例都將在一個小型標準機器學習資料集上進行演示，該資料集是一個糖尿病發作分類資料集。這是一個小型資料集，所有的數值屬性都很容易處理。
在第一個簡單範例中，我們將介紹如何調優批次大小和擬合網路時使用的epoch數。
我們將簡單評估從10到100的不批大小，程式碼清單如下所示:
import random import numpy as np import torch import torch.nn as nn import torch.optim as optim from skorch import NeuralNetClassifier from sklearn.model_selection import GridSearchCV # load the dataset, split into input (X) and output (y) variables dataset = np.loadtxt('pima-indians-diabetes.csv', delimiter=',') X = dataset[:,0:8] y = dataset[:,8] X = torch.tensor(X, dtype=torch.float32) y = torch.tensor(y, dtype=torch.float32).reshape(-1, 1) # PyTorch classifier class PimaClassifier(nn.Module): def __init__(self): super().__init__() self.layer = nn.Linear(8, 12) self.act = nn.ReLU() self.output = nn.Linear(12, 1) self.prob = nn.Sigmoid() def forward(self, x): x = self.act(self.layer(x)) x = self.prob(self.output(x)) return x # create model with skorch model = NeuralNetClassifier( PimaClassifier, criterinotallow=nn.BCELoss, optimizer=optim.Adam, verbose=False ) # define the grid search parameters param_grid = { 'batch_size': [10, 20, 40, 60, 80, 100], 'max_epochs': [10, 50, 100] } grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3) grid_result = grid.fit(X, y) # summarize results print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_)) means = grid_result.cv_results_['mean_test_score'] stds = grid_result.cv_results_['std_test_score'] params = grid_result.cv_results_['params'] for mean, stdev, param in zip(means, stds, params): print("%f (%f) with: %r" % (mean, stdev, param))
結果如下:
Best: 0.714844 using {'batch_size': 10, 'max_epochs': 100} 0.665365 (0.020505) with: {'batch_size': 10, 'max_epochs': 10} 0.588542 (0.168055) with: {'batch_size': 10, 'max_epochs': 50} 0.714844 (0.032369) with: {'batch_size': 10, 'max_epochs': 100} 0.671875 (0.022326) with: {'batch_size': 20, 'max_epochs': 10} 0.696615 (0.008027) with: {'batch_size': 20, 'max_epochs': 50} 0.714844 (0.019918) with: {'batch_size': 20, 'max_epochs': 100} 0.666667 (0.009744) with: {'batch_size': 40, 'max_epochs': 10} 0.687500 (0.033603) with: {'batch_size': 40, 'max_epochs': 50} 0.707031 (0.024910) with: {'batch_size': 40, 'max_epochs': 100} 0.667969 (0.014616) with: {'batch_size': 60, 'max_epochs': 10} 0.694010 (0.036966) with: {'batch_size': 60, 'max_epochs': 50} 0.694010 (0.042473) with: {'batch_size': 60, 'max_epochs': 100} 0.670573 (0.023939) with: {'batch_size': 80, 'max_epochs': 10} 0.674479 (0.020752) with: {'batch_size': 80, 'max_epochs': 50} 0.703125 (0.026107) with: {'batch_size': 80, 'max_epochs': 100} 0.680990 (0.014382) with: {'batch_size': 100, 'max_epochs': 10} 0.670573 (0.013279) with: {'batch_size': 100, 'max_epochs': 50} 0.687500 (0.017758) with: {'batch_size': 100, 'max_epochs': 100}
可以看到'batch_size': 10 , 'max_epochs': 100達到了約71%的精度的最佳結果。
下面我們來看看如何調整最佳化器，我們知道有很多個最佳化器可以選擇像是SDG,Adam等，那麼如何選擇呢？
完整的程式碼如下：
import numpy as np import torch import torch.nn as nn import torch.optim as optim from skorch import NeuralNetClassifier from sklearn.model_selection import GridSearchCV # load the dataset, split into input (X) and output (y) variables dataset = np.loadtxt('pima-indians-diabetes.csv', delimiter=',') X = dataset[:,0:8] y = dataset[:,8] X = torch.tensor(X, dtype=torch.float32) y = torch.tensor(y, dtype=torch.float32).reshape(-1, 1) # PyTorch classifier class PimaClassifier(nn.Module): def __init__(self): super().__init__() self.layer = nn.Linear(8, 12) self.act = nn.ReLU() self.output = nn.Linear(12, 1) self.prob = nn.Sigmoid() def forward(self, x): x = self.act(self.layer(x)) x = self.prob(self.output(x)) return x # create model with skorch model = NeuralNetClassifier( PimaClassifier, criterinotallow=nn.BCELoss, max_epochs=100, batch_size=10, verbose=False ) # define the grid search parameters param_grid = { 'optimizer': [optim.SGD, optim.RMSprop, optim.Adagrad, optim.Adadelta, optim.Adam, optim.Adamax, optim.NAdam], } grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3) grid_result = grid.fit(X, y) # summarize results print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_)) means = grid_result.cv_results_['mean_test_score'] stds = grid_result.cv_results_['std_test_score'] params = grid_result.cv_results_['params'] for mean, stdev, param in zip(means, stds, params): print("%f (%f) with: %r" % (mean, stdev, param))
輸出如下：
Best: 0.721354 using {'optimizer': } 0.674479 (0.036828) with: {'optimizer': } 0.700521 (0.043303) with: {'optimizer': } 0.682292 (0.027126) with: {'optimizer': } 0.572917 (0.051560) with: {'optimizer': } 0.714844 (0.030758) with: {'optimizer': } 0.721354 (0.019225) with: {'optimizer': } 0.709635 (0.024360) with: {'optimizer': }
#可以看到對於我們的模型和資料集Adamax最佳化演算法是最佳的，準確率約為72%。
雖然pytorch裡面學習率計畫可以讓我們根據輪次動態調整學習率，但是作為範例，我們將學習率和學習率的參數作為網格搜尋的一個參數來進行示範。在PyTorch中，設定學習率和動量的方法如下:
optimizer = optim.SGD(lr=0.001, momentum=0.9)
在skorch套件中，使用前綴optimizer__將參數路由到優化器。
import numpy as np import torch import torch.nn as nn import torch.optim as optim from skorch import NeuralNetClassifier from sklearn.model_selection import GridSearchCV # load the dataset, split into input (X) and output (y) variables dataset = np.loadtxt('pima-indians-diabetes.csv', delimiter=',') X = dataset[:,0:8] y = dataset[:,8] X = torch.tensor(X, dtype=torch.float32) y = torch.tensor(y, dtype=torch.float32).reshape(-1, 1) # PyTorch classifier class PimaClassifier(nn.Module): def __init__(self): super().__init__() self.layer = nn.Linear(8, 12) self.act = nn.ReLU() self.output = nn.Linear(12, 1) self.prob = nn.Sigmoid() def forward(self, x): x = self.act(self.layer(x)) x = self.prob(self.output(x)) return x # create model with skorch model = NeuralNetClassifier( PimaClassifier, criterinotallow=nn.BCELoss, optimizer=optim.SGD, max_epochs=100, batch_size=10, verbose=False ) # define the grid search parameters param_grid = { 'optimizer__lr': [0.001, 0.01, 0.1, 0.2, 0.3], 'optimizer__momentum': [0.0, 0.2, 0.4, 0.6, 0.8, 0.9], } grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3) grid_result = grid.fit(X, y) # summarize results print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_)) means = grid_result.cv_results_['mean_test_score'] stds = grid_result.cv_results_['std_test_score'] params = grid_result.cv_results_['params'] for mean, stdev, param in zip(means, stds, params): print("%f (%f) with: %r" % (mean, stdev, param))
結果如下：
Best: 0.682292 using {'optimizer__lr': 0.001, 'optimizer__momentum': 0.9} 0.648438 (0.016877) with: {'optimizer__lr': 0.001, 'optimizer__momentum': 0.0} 0.671875 (0.017758) with: {'optimizer__lr': 0.001, 'optimizer__momentum': 0.2} 0.674479 (0.022402) with: {'optimizer__lr': 0.001, 'optimizer__momentum': 0.4} 0.677083 (0.011201) with: {'optimizer__lr': 0.001, 'optimizer__momentum': 0.6} 0.679688 (0.027621) with: {'optimizer__lr': 0.001, 'optimizer__momentum': 0.8} 0.682292 (0.026557) with: {'optimizer__lr': 0.001, 'optimizer__momentum': 0.9} 0.671875 (0.019918) with: {'optimizer__lr': 0.01, 'optimizer__momentum': 0.0} 0.648438 (0.024910) with: {'optimizer__lr': 0.01, 'optimizer__momentum': 0.2} 0.546875 (0.143454) with: {'optimizer__lr': 0.01, 'optimizer__momentum': 0.4} 0.567708 (0.153668) with: {'optimizer__lr': 0.01, 'optimizer__momentum': 0.6} 0.552083 (0.141790) with: {'optimizer__lr': 0.01, 'optimizer__momentum': 0.8} 0.451823 (0.144561) with: {'optimizer__lr': 0.01, 'optimizer__momentum': 0.9} 0.348958 (0.001841) with: {'optimizer__lr': 0.1, 'optimizer__momentum': 0.0} 0.450521 (0.142719) with: {'optimizer__lr': 0.1, 'optimizer__momentum': 0.2} 0.450521 (0.142719) with: {'optimizer__lr': 0.1, 'optimizer__momentum': 0.4} 0.450521 (0.142719) with: {'optimizer__lr': 0.1, 'optimizer__momentum': 0.6} 0.348958 (0.001841) with: {'optimizer__lr': 0.1, 'optimizer__momentum': 0.8} 0.348958 (0.001841) with: {'optimizer__lr': 0.1, 'optimizer__momentum': 0.9} 0.444010 (0.136265) with: {'optimizer__lr': 0.2, 'optimizer__momentum': 0.0} 0.450521 (0.142719) with: {'optimizer__lr': 0.2, 'optimizer__momentum': 0.2} 0.348958 (0.001841) with: {'optimizer__lr': 0.2, 'optimizer__momentum': 0.4} 0.552083 (0.141790) with: {'optimizer__lr': 0.2, 'optimizer__momentum': 0.6} 0.549479 (0.142719) with: {'optimizer__lr': 0.2, 'optimizer__momentum': 0.8} 0.651042 (0.001841) with: {'optimizer__lr': 0.2, 'optimizer__momentum': 0.9} 0.552083 (0.141790) with: {'optimizer__lr': 0.3, 'optimizer__momentum': 0.0} 0.348958 (0.001841) with: {'optimizer__lr': 0.3, 'optimizer__momentum': 0.2} 0.450521 (0.142719) with: {'optimizer__lr': 0.3, 'optimizer__momentum': 0.4} 0.552083 (0.141790) with: {'optimizer__lr': 0.3, 'optimizer__momentum': 0.6} 0.450521 (0.142719) with: {'optimizer__lr': 0.3, 'optimizer__momentum': 0.8} 0.450521 (0.142719) with: {'optimizer__lr': 0.3, 'optimizer__momentum': 0.9}
對於SGD，使用0.001的學習率和0.9的動量獲得了最佳結果，準確率約為68%。
激活函數控制單一神經元的非線性。我們將示範評估PyTorch中可用的一些激活函數。
import numpy as np import torch import torch.nn as nn import torch.nn.init as init import torch.optim as optim from skorch import NeuralNetClassifier from sklearn.model_selection import GridSearchCV # load the dataset, split into input (X) and output (y) variables dataset = np.loadtxt('pima-indians-diabetes.csv', delimiter=',') X = dataset[:,0:8] y = dataset[:,8] X = torch.tensor(X, dtype=torch.float32) y = torch.tensor(y, dtype=torch.float32).reshape(-1, 1) # PyTorch classifier class PimaClassifier(nn.Module): def __init__(self, activatinotallow=nn.ReLU): super().__init__() self.layer = nn.Linear(8, 12) self.act = activation() self.output = nn.Linear(12, 1) self.prob = nn.Sigmoid() # manually init weights init.kaiming_uniform_(self.layer.weight) init.kaiming_uniform_(self.output.weight) def forward(self, x): x = self.act(self.layer(x)) x = self.prob(self.output(x)) return x # create model with skorch model = NeuralNetClassifier( PimaClassifier, criterinotallow=nn.BCELoss, optimizer=optim.Adamax, max_epochs=100, batch_size=10, verbose=False ) # define the grid search parameters param_grid = { 'module__activation': [nn.Identity, nn.ReLU, nn.ELU, nn.ReLU6, nn.GELU, nn.Softplus, nn.Softsign, nn.Tanh, nn.Sigmoid, nn.Hardsigmoid] } grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3) grid_result = grid.fit(X, y) # summarize results print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_)) means = grid_result.cv_results_['mean_test_score'] stds = grid_result.cv_results_['std_test_score'] params = grid_result.cv_results_['params'] for mean, stdev, param in zip(means, stds, params): print("%f (%f) with: %r" % (mean, stdev, param))
結果如下：
Best: 0.699219 using {'module__activation': } 0.687500 (0.025315) with: {'module__activation': } 0.699219 (0.011049) with: {'module__activation': } 0.674479 (0.035849) with: {'module__activation': } 0.621094 (0.063549) with: {'module__activation': } 0.674479 (0.017566) with: {'module__activation': } 0.558594 (0.149189) with: {'module__activation': } 0.675781 (0.014616) with: {'module__activation': } 0.619792 (0.018688) with: {'module__activation': } 0.643229 (0.019225) with: {'module__activation': } 0.636719 (0.022326) with: {'module__activation': }
ReLU激活函數獲得了最好的結果，準確率約為70%。
在本例中，我们将尝试在0.0到0.9之间的dropout百分比(1.0没有意义)和在0到5之间的MaxNorm权重约束值。
import numpy as np import torch import torch.nn as nn import torch.nn.init as init import torch.optim as optim from skorch import NeuralNetClassifier from sklearn.model_selection import GridSearchCV # load the dataset, split into input (X) and output (y) variables dataset = np.loadtxt('pima-indians-diabetes.csv', delimiter=',') X = dataset[:,0:8] y = dataset[:,8] X = torch.tensor(X, dtype=torch.float32) y = torch.tensor(y, dtype=torch.float32).reshape(-1, 1) # PyTorch classifier class PimaClassifier(nn.Module): def __init__(self, dropout_rate=0.5, weight_cnotallow=1.0): super().__init__() self.layer = nn.Linear(8, 12) self.act = nn.ReLU() self.dropout = nn.Dropout(dropout_rate) self.output = nn.Linear(12, 1) self.prob = nn.Sigmoid() self.weight_constraint = weight_constraint # manually init weights init.kaiming_uniform_(self.layer.weight) init.kaiming_uniform_(self.output.weight) def forward(self, x): # maxnorm weight before actual forward pass with torch.no_grad(): norm = self.layer.weight.norm(2, dim=0, keepdim=True).clamp(min=self.weight_constraint / 2) desired = torch.clamp(norm, max=self.weight_constraint) self.layer.weight *= (desired / norm) # actual forward pass x = self.act(self.layer(x)) x = self.dropout(x) x = self.prob(self.output(x)) return x # create model with skorch model = NeuralNetClassifier( PimaClassifier, criterinotallow=nn.BCELoss, optimizer=optim.Adamax, max_epochs=100, batch_size=10, verbose=False ) # define the grid search parameters param_grid = { 'module__weight_constraint': [1.0, 2.0, 3.0, 4.0, 5.0], 'module__dropout_rate': [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9] } grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3) grid_result = grid.fit(X, y) # summarize results print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_)) means = grid_result.cv_results_['mean_test_score'] stds = grid_result.cv_results_['std_test_score'] params = grid_result.cv_results_['params'] for mean, stdev, param in zip(means, stds, params): print("%f (%f) with: %r" % (mean, stdev, param))
结果如下：
Best: 0.701823 using {'module__dropout_rate': 0.1, 'module__weight_constraint': 2.0} 0.669271 (0.015073) with: {'module__dropout_rate': 0.0, 'module__weight_constraint': 1.0} 0.692708 (0.035132) with: {'module__dropout_rate': 0.0, 'module__weight_constraint': 2.0} 0.589844 (0.170180) with: {'module__dropout_rate': 0.0, 'module__weight_constraint': 3.0} 0.561198 (0.151131) with: {'module__dropout_rate': 0.0, 'module__weight_constraint': 4.0} 0.688802 (0.021710) with: {'module__dropout_rate': 0.0, 'module__weight_constraint': 5.0} 0.697917 (0.009744) with: {'module__dropout_rate': 0.1, 'module__weight_constraint': 1.0} 0.701823 (0.016367) with: {'module__dropout_rate': 0.1, 'module__weight_constraint': 2.0} 0.694010 (0.010253) with: {'module__dropout_rate': 0.1, 'module__weight_constraint': 3.0} 0.686198 (0.025976) with: {'module__dropout_rate': 0.1, 'module__weight_constraint': 4.0} 0.679688 (0.026107) with: {'module__dropout_rate': 0.1, 'module__weight_constraint': 5.0} 0.701823 (0.029635) with: {'module__dropout_rate': 0.2, 'module__weight_constraint': 1.0} 0.682292 (0.014731) with: {'module__dropout_rate': 0.2, 'module__weight_constraint': 2.0} 0.701823 (0.009744) with: {'module__dropout_rate': 0.2, 'module__weight_constraint': 3.0} 0.701823 (0.026557) with: {'module__dropout_rate': 0.2, 'module__weight_constraint': 4.0} 0.687500 (0.015947) with: {'module__dropout_rate': 0.2, 'module__weight_constraint': 5.0} 0.686198 (0.006639) with: {'module__dropout_rate': 0.3, 'module__weight_constraint': 1.0} 0.656250 (0.006379) with: {'module__dropout_rate': 0.3, 'module__weight_constraint': 2.0} 0.565104 (0.155608) with: {'module__dropout_rate': 0.3, 'module__weight_constraint': 3.0} 0.700521 (0.028940) with: {'module__dropout_rate': 0.3, 'module__weight_constraint': 4.0} 0.669271 (0.012890) with: {'module__dropout_rate': 0.3, 'module__weight_constraint': 5.0} 0.661458 (0.018688) with: {'module__dropout_rate': 0.4, 'module__weight_constraint': 1.0} 0.669271 (0.017566) with: {'module__dropout_rate': 0.4, 'module__weight_constraint': 2.0} 0.652344 (0.006379) with: {'module__dropout_rate': 0.4, 'module__weight_constraint': 3.0} 0.680990 (0.037783) with: {'module__dropout_rate': 0.4, 'module__weight_constraint': 4.0} 0.692708 (0.042112) with: {'module__dropout_rate': 0.4, 'module__weight_constraint': 5.0} 0.666667 (0.006639) with: {'module__dropout_rate': 0.5, 'module__weight_constraint': 1.0} 0.652344 (0.011500) with: {'module__dropout_rate': 0.5, 'module__weight_constraint': 2.0} 0.662760 (0.007366) with: {'module__dropout_rate': 0.5, 'module__weight_constraint': 3.0} 0.558594 (0.146610) with: {'module__dropout_rate': 0.5, 'module__weight_constraint': 4.0} 0.552083 (0.141826) with: {'module__dropout_rate': 0.5, 'module__weight_constraint': 5.0} 0.548177 (0.141826) with: {'module__dropout_rate': 0.6, 'module__weight_constraint': 1.0} 0.653646 (0.013279) with: {'module__dropout_rate': 0.6, 'module__weight_constraint': 2.0} 0.661458 (0.008027) with: {'module__dropout_rate': 0.6, 'module__weight_constraint': 3.0} 0.553385 (0.142719) with: {'module__dropout_rate': 0.6, 'module__weight_constraint': 4.0} 0.669271 (0.035132) with: {'module__dropout_rate': 0.6, 'module__weight_constraint': 5.0} 0.662760 (0.015733) with: {'module__dropout_rate': 0.7, 'module__weight_constraint': 1.0} 0.636719 (0.024910) with: {'module__dropout_rate': 0.7, 'module__weight_constraint': 2.0} 0.550781 (0.146818) with: {'module__dropout_rate': 0.7, 'module__weight_constraint': 3.0} 0.537760 (0.140094) with: {'module__dropout_rate': 0.7, 'module__weight_constraint': 4.0} 0.542969 (0.138144) with: {'module__dropout_rate': 0.7, 'module__weight_constraint': 5.0} 0.565104 (0.148654) with: {'module__dropout_rate': 0.8, 'module__weight_constraint': 1.0} 0.657552 (0.008027) with: {'module__dropout_rate': 0.8, 'module__weight_constraint': 2.0} 0.428385 (0.111418) with: {'module__dropout_rate': 0.8, 'module__weight_constraint': 3.0} 0.549479 (0.142719) with: {'module__dropout_rate': 0.8, 'module__weight_constraint': 4.0} 0.648438 (0.005524) with: {'module__dropout_rate': 0.8, 'module__weight_constraint': 5.0} 0.540365 (0.136861) with: {'module__dropout_rate': 0.9, 'module__weight_constraint': 1.0} 0.605469 (0.053083) with: {'module__dropout_rate': 0.9, 'module__weight_constraint': 2.0} 0.553385 (0.139948) with: {'module__dropout_rate': 0.9, 'module__weight_constraint': 3.0} 0.549479 (0.142719) with: {'module__dropout_rate': 0.9, 'module__weight_constraint': 4.0} 0.595052 (0.075566) with: {'module__dropout_rate': 0.9, 'module__weight_constraint': 5.0}
可以看到，10%的Dropout和2.0的权重约束获得了70%的最佳精度。
单层神经元的数量是一个需要调优的重要参数。一般来说，一层神经元的数量控制着网络的表示能力，至少在拓扑的这一点上是这样。
理论上来说：由于通用逼近定理，一个足够大的单层网络可以近似任何其他神经网络。
在本例中，将尝试从1到30的值，步骤为5。一个更大的网络需要更多的训练，至少批大小和epoch的数量应该与神经元的数量一起优化。
import numpy as np import torch import torch.nn as nn import torch.nn.init as init import torch.optim as optim from skorch import NeuralNetClassifier from sklearn.model_selection import GridSearchCV # load the dataset, split into input (X) and output (y) variables dataset = np.loadtxt('pima-indians-diabetes.csv', delimiter=',') X = dataset[:,0:8] y = dataset[:,8] X = torch.tensor(X, dtype=torch.float32) y = torch.tensor(y, dtype=torch.float32).reshape(-1, 1) class PimaClassifier(nn.Module): def __init__(self, n_neurnotallow=12): super().__init__() self.layer = nn.Linear(8, n_neurons) self.act = nn.ReLU() self.dropout = nn.Dropout(0.1) self.output = nn.Linear(n_neurons, 1) self.prob = nn.Sigmoid() self.weight_constraint = 2.0 # manually init weights init.kaiming_uniform_(self.layer.weight) init.kaiming_uniform_(self.output.weight) def forward(self, x): # maxnorm weight before actual forward pass with torch.no_grad(): norm = self.layer.weight.norm(2, dim=0, keepdim=True).clamp(min=self.weight_constraint / 2) desired = torch.clamp(norm, max=self.weight_constraint) self.layer.weight *= (desired / norm) # actual forward pass x = self.act(self.layer(x)) x = self.dropout(x) x = self.prob(self.output(x)) return x # create model with skorch model = NeuralNetClassifier( PimaClassifier, criterinotallow=nn.BCELoss, optimizer=optim.Adamax, max_epochs=100, batch_size=10, verbose=False ) # define the grid search parameters param_grid = { 'module__n_neurons': [1, 5, 10, 15, 20, 25, 30] } grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3) grid_result = grid.fit(X, y) # summarize results print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_)) means = grid_result.cv_results_['mean_test_score'] stds = grid_result.cv_results_['std_test_score'] params = grid_result.cv_results_['params'] for mean, stdev, param in zip(means, stds, params): print("%f (%f) with: %r" % (mean, stdev, param))
结果如下：
Best: 0.708333 using {'module__n_neurons': 30} 0.654948 (0.003683) with: {'module__n_neurons': 1} 0.666667 (0.023073) with: {'module__n_neurons': 5} 0.694010 (0.014382) with: {'module__n_neurons': 10} 0.682292 (0.014382) with: {'module__n_neurons': 15} 0.707031 (0.028705) with: {'module__n_neurons': 20} 0.703125 (0.030758) with: {'module__n_neurons': 25} 0.708333 (0.015733) with: {'module__n_neurons': 30}
你可以看到，在隐藏层中有30个神经元的网络获得了最好的结果，准确率约为71%。
以上是如何使用scikit-learn為PyTorch模型進行超參數網格搜尋？的詳細內容。更多資訊請關注PHP中文網其他相關文章！
