In this tutorial, we will discuss the Grid Search for the reason of hyperparameter tuning. We will also analyze about the working of Grid Search alongside with the implementation of it in optimizing the overall performance of the technique of Machine Learning (ML).
Hyperparameter tuning is sizable for the excellent working of the fashions of Machine Learning (ML). A technique like Grid Search seems to be a simple utility for hyperparameter optimization.
The Grid Search Method considers some hyperparameter combos and selects the one returning a decrease error score. This method is mainly useful when there are solely some hyperparameters in order to optimize. However, it is outperformed via different weighted-random search strategies when the Machine Learning model grows in complexity.
So let us begin by understanding Grid Search.
Understanding Grid Search
Grid Search is an optimization algorithm that allows us to pick out the pleasant parameters to optimize the trouble from a listing of parameter alternatives we are providing, for that reason automating the ‘trial-and-error’ method. Although we can apply it to a couple of optimization issues; however, it is most in many instances recognised for its utilization in laptop studying in order to achieve the parameters at which the model gives the satisfactory accuracy.
Let us consider that the model accepts the beneath three parameters in the structure of input:
Number of hidden layers [2, 4]
Number of neurons in every layer [5, 10]
Number of epochs [10, 50]
If we choose to strive out two picks for each parameter input (as precise in square brackets above), it estimates different combinations. For instance, one viable aggregate can be [2, 5, 10]. Finding such combinations manually would be a headache.
Now, feel that we had ten distinct parameters as input, and we would like to try out 5 feasible values for each and every parameter. It would need guide input from the programmer’s quit each time we like to alter the cost of a parameter, re-execute the code, and maintain a document of the outputs for each mixture of the parameters.
Grid Search automates that process, as it accepts the possible cost for each parameter and executes the code in order to strive out every and every viable combination outputs the end result for the combinations and outputs the aggregate having the first-class accuracy.
Installing the required libraries
Before we start enforcing the Grid Search in the Python programming language, let us temporarily talk about some of the quintessential libraries and frameworks that want to be installed in the system.
These libraries and frameworks are as follows:
Python 3
NumPy
Pandas
Keras
Scikit-Learn
They are all pretty easy to install. We can use the pip installer in order to deploy these libraries as proven below:
$ pip install numpy tensorflow pandas scikit-learn keras
Note: If any troubles arise while executing any package, strive reinstalling and referring to each package’s authentic documentation.
Now, let us start enforcing the Grid Search in Python
Implementation of Grid Search in Python
In the following section, we will understand how to put into effect Grid Search on an proper application. We will genuinely be executing the code and discuss in-depth concerning the part the place Grid Search comes in alternatively than discussing Machine Learning and Data Pre-processing section.
So, let’s get started.
We will use the Diet Dataset containing statistics concerning the peak and weight of exceptional humans based totally on a number of attributes such as gender, age, and kind of diet. We can immediately import the records from an on line aid with the help of the Pandas read_csv() function.
But before that, we have to import the required packages:
File: mygrid.py
import sys
import pandas as pd
import numpy as np
from sklearn.model_selection import GridSearchCV, KFold
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.wrappers.scikit_learn import KerasClassifier
from keras.optimizers import Adam
Explanation:
In the above snippet of code, we have imported the required applications and libraries integral for the project. One can also retailer the software file and execute it in order to take a look at if the libraries and applications are hooked up and imported properly.
Once the packages are imported successfully, we have to use the following snippet of code in order to import the dataset and print the first 5 rows of it.
File: mygrid.py
# importing the dataset
mydf = pd.read_csv("Diet_Dataset.csv")
# printing the first five lines of dataset
print(mydf.head())
Output:
Person gender Age Height pre.weight Diet weight6weeks
0 25 41 171 60 2 60.0
1 26 32 174 103 2 103.0
2 1 0 22 159 58 1 54.2
3 2 0 46 192 60 1 54.0
4 3 0 55 170 64 1 63.3Explanation:
In the above snippet of code, we have imported the dataset the usage of the read_csv() of the pandas library and saved it inside the mydf variable. We have then printed the first five rows the use of the head() feature along with the mydf variable.
Now, let us divide the information into the function and label units in order to practice the general scaling on the dataset.
The snippet of code for the same is shown below:
File: mygrid.py
# converting dataframe into numpy array
mydataset = mydf.values
X = mydataset[:, 0:6]
Y = mydataset[:, 6].astype(int)
# Normalizing the data using sklearn StandardScaler
from sklearn.preprocessing import StandardScaler
myscaler = StandardScaler().fit(X)
# Transforming and displaying the training data
X_stdized = myscaler.transform(X)
mydata = pd.DataFrame(X_stdized)
Explanation:
In the above snippet of code, we have transformed the pandas dataframe into a NumPy array. We have then imported the StandardScaler module from the sklearn library and use the function to normalize the data. We have then converted and displayed the coaching data the use of the transform() function.
Now, let us reflect onconsideration on the following snippet of code in order to create a easy deep gaining knowledge of model.
File: mygrid.py
# defining the function to create model
def create_my_model(learnRate, dropoutRate):
# Creating model
mymodel = Sequential()
mymodel.add(Dense(6, input_dim = 6, kernel_initializer = 'normal', activation = 'relu'))
mymodel.add(Dropout(dropoutRate))
mymodel.add(Dense(3, input_dim = 6, kernel_initializer = 'normal', activation = 'relu'))
mymodel.add(Dropout(dropoutRate))
mymodel.add(Dense(1, activation = 'sigmoid'))
# Compiling the model
my_Adam = Adam(learning_rate = learnRate)
mymodel.compile(loss = 'binary_crossentropy', optimizer = my_Adam, metrics = ['accuracy'])
return mymodel
Explanation:
The following snippet of code has described a function as create_my_model() accepting two parameters, i.e., learnRate and dropoutRate, respectively. We have then created the model as mymodel the usage of the Sequential() function. We have additionally used the add() along with the Dense() and Dropout() function. We have then compiled the model using the compile() function.
As a result, when we execute the code, this will lead to loading the dataset, preprocessing it, and creating a machine studying model. Since we are solely involved in grasp the working of Grid Search, we haven’t carried out the train/test split, and we had fitted the model on the complete dataset.
Now, we will recognize how Grid Search makes the programmer’s existence less difficult through optimizing the parameters in the next section.
Training the Model without Grid Search
In the snippet of code proven below, we will create a model with the assist of parameter values that we decided on randomly or based totally on our instinct and see how our model performs:
File: mygrid.py
# Declaring the values of the parameter
dropoutRate = 0.1
epochs = 1
batchSize = 20
learnRate = 0.001
# Creating the model object by calling the create_my_model function we created above
mymodel = create_my_model(learnRate, dropoutRate)
# Fitting the model onto the training data
mymodel.fit(X_stdized, Y, batch_size = batchSize, epochs = epochs, verbose = 1)
Output:
4/4 [==============================] - 41s 14ms/step - loss: 0.9364 - accuracy: 0.0000e+00Explanation:
In the above snippet of code, we have declared the values of the parameter, i.e., dropoutRate, epochs, batchSize, and learnRate, respectively. We have then created the mannequin object by way of calling the create_my_model() function. We have then equipped the mannequin onto the training data.
As a result, the accuracy we got is 0.0000e+00.
Optimizing Hyper-parameters using Grid Search
If we are no longer using the Grid Search method, we can without delay name the fit() feature on the model we have created above. But in order to utilize the Grid Search method, we have to ignore in few arguments to the create_my_model() function. Moreover, we have to declare the grid with a number preferences to try for each parameter. Let us perform that in exceptional parts.
First of all, we will try editing the create_my_model() function in order to accept arguments from the calling function as shown below:
File: mygrid.py
def create_my_model(learnRate, dropoutRate):
# Creating the model
mymodel = Sequential()
mymodel.add(Dense(6, input_dim = 6, kernel_initializer = 'normal', activation = 'relu'))
mymodel.add(Dropout(dropoutRate))
mymodel.add(Dense(3, input_dim = 6, kernel_initializer = 'normal', activation = 'relu'))
mymodel.add(Dropout(dropoutRate))
mymodel.add(Dense(1, activation = 'sigmoid'))
# Compile the model
myadam = Adam(learning_rate = learnRate)
mymodel.compile(loss = 'binary_crossentropy', optimizer = myadam, metrics = ['accuracy'])
return mymodel
# Creating the model object
mymodel = KerasClassifier(build_fn = create_my_model, verbose = 1)
Explanation:
In the above snippet of code, we have made some changes to the preceding create_my_model feature and used the KerasClassifier to create the model object.
Now, let us put in force the algorithm for Grid Search and in shape the dataset on it:
File: mygrid.py
# Defining the arguments that we want to use in Grid Search along
# with the list of values that we want to try out
learnRate = [0.001, 0.02, 0.2]
dropoutRate = [0.0, 0.2, 0.4]
batchSize = [10, 20, 30]
epochs = [1, 5, 10]
# Making a dictionary of the grid search parameters
paramgrid = dict(learnRate = learnRate, dropoutRate = dropoutRate, batch_size = batchSize, epochs = epochs )
# Building and fitting the GridSearchCV
mygrid = GridSearchCV(estimator = mymodel, param_grid = paramgrid, cv = KFold(random_state = None), verbose = 10)
gridresults = mygrid.fit(X_stdized, Y)
# Summarizing the results in a readable format
print("Best: " + gridresults.best_score_ + " using " + gridresults.best_params_)
means = gridresults.cv_results_['mean_test_score']
stds = gridresults.cv_results_['std_test_score']
params = gridresults.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
print(mean + "(" + stdev + ")" + " with: " + param)
Output:
Best: 0.00347268912077, using {batch_size=10, dropoutRate=0.4, epochs=5, learnRate=0.2}Explanation:
The above output shows the parameter combination which yields the quality accuracy.
At last, we can conclude that the Grid Search is quite easy to implement in the Python programming language and saved us a lot of time in human labor. We can listing down all the arguments we wanted to tune, declare the values that need to be tested, execute the code, and neglect about it. The manner is so convenient and convenient that it requires much less input from the programmer’s side. Once the fine argument combination has been found, we can utilize that for the last model.