Learn Python – Grid Search in Python- Basic and advance

In this tutorial, we will discuss the Grid Search for the reason of hyperparameter tuning. We will also analyze about the working of Grid Search alongside with the implementation of it in optimizing the overall performance of the technique of Machine Learning (ML).

Hyperparameter tuning is sizable for the excellent working of the fashions of Machine Learning (ML). A technique like Grid Search seems to be a simple utility for hyperparameter optimization.

The Grid Search Method considers some hyperparameter combos and selects the one returning a decrease error score. This method is mainly useful when there are solely some hyperparameters in order to optimize. However, it is outperformed via different weighted-random search strategies when the Machine Learning model grows in complexity.

So let us begin by understanding Grid Search.

Understanding Grid Search

Grid Search is an optimization algorithm that allows us to pick out the pleasant parameters to optimize the trouble from a listing of parameter alternatives we are providing, for that reason automating the ‘trial-and-error’ method. Although we can apply it to a couple of optimization issues; however, it is most in many instances recognised for its utilization in laptop studying in order to achieve the parameters at which the model gives the satisfactory accuracy.

Let us consider that the model accepts the beneath three parameters in the structure of input:

Number of hidden layers [2, 4]
Number of neurons in every layer [5, 10]
Number of epochs [10, 50]

If we choose to strive out two picks for each parameter input (as precise in square brackets above), it estimates different combinations. For instance, one viable aggregate can be [2, 5, 10]. Finding such combinations manually would be a headache.

Now, feel that we had ten distinct parameters as input, and we would like to try out 5 feasible values for each and every parameter. It would need guide input from the programmer’s quit each time we like to alter the cost of a parameter, re-execute the code, and maintain a document of the outputs for each mixture of the parameters.

Grid Search automates that process, as it accepts the possible cost for each parameter and executes the code in order to strive out every and every viable combination outputs the end result for the combinations and outputs the aggregate having the first-class accuracy.

Installing the required libraries

Before we start enforcing the Grid Search in the Python programming language, let us temporarily talk about some of the quintessential libraries and frameworks that want to be installed in the system.

These libraries and frameworks are as follows:

Python 3
NumPy
Pandas
Keras
Scikit-Learn

They are all pretty easy to install. We can use the pip installer in order to deploy these libraries as proven below:

$ pip install numpy tensorflow pandas scikit-learn keras  

Note: If any troubles arise while executing any package, strive reinstalling and referring to each package’s authentic documentation.

Now, let us start enforcing the Grid Search in Python

Implementation of Grid Search in Python

In the following section, we will understand how to put into effect Grid Search on an proper application. We will genuinely be executing the code and discuss in-depth concerning the part the place Grid Search comes in alternatively than discussing Machine Learning and Data Pre-processing section.

So, let’s get started.

We will use the Diet Dataset containing statistics concerning the peak and weight of exceptional humans based totally on a number of attributes such as gender, age, and kind of diet. We can immediately import the records from an on line aid with the help of the Pandas read_csv() function.

But before that, we have to import the required packages:

File: mygrid.py

import sys  
import pandas as pd  
import numpy as np  
from sklearn.model_selection import GridSearchCV, KFold  
from keras.models import Sequential  
from keras.layers import Dense, Dropout  
from keras.wrappers.scikit_learn import KerasClassifier  
from keras.optimizers import Adam  

Explanation:

In the above snippet of code, we have imported the required applications and libraries integral for the project. One can also retailer the software file and execute it in order to take a look at if the libraries and applications are hooked up and imported properly.

Once the packages are imported successfully, we have to use the following snippet of code in order to import the dataset and print the first 5 rows of it.

File: mygrid.py

# importing the dataset  
mydf = pd.read_csv("Diet_Dataset.csv")  
  
# printing the first five lines of dataset  
print(mydf.head())  

Output:

   Person gender  Age  Height  pre.weight  Diet  weight6weeks
0      25          41     171          60     2          60.0
1      26          32     174         103     2         103.0
2       1      0   22     159          58     1          54.2
3       2      0   46     192          60     1          54.0
4       3      0   55     170          64     1          63.3

Explanation:

In the above snippet of code, we have imported the dataset the usage of the read_csv() of the pandas library and saved it inside the mydf variable. We have then printed the first five rows the use of the head() feature along with the mydf variable.

Now, let us divide the information into the function and label units in order to practice the general scaling on the dataset.

The snippet of code for the same is shown below:

File: mygrid.py

# converting dataframe into numpy array  
mydataset = mydf.values  
  
X = mydataset[:, 0:6]  
Y = mydataset[:, 6].astype(int)  
  
# Normalizing the data using sklearn StandardScaler  
from sklearn.preprocessing import StandardScaler  
  
myscaler = StandardScaler().fit(X)  
  
# Transforming and displaying the training data  
X_stdized = myscaler.transform(X)  
  
mydata = pd.DataFrame(X_stdized)  

Explanation:

In the above snippet of code, we have transformed the pandas dataframe into a NumPy array. We have then imported the StandardScaler module from the sklearn library and use the function to normalize the data. We have then converted and displayed the coaching data the use of the transform() function.

Now, let us reflect onconsideration on the following snippet of code in order to create a easy deep gaining knowledge of model.

File: mygrid.py

# defining the function to create model  
def create_my_model(learnRate, dropoutRate):  
    # Creating model  
    mymodel = Sequential()  
    mymodel.add(Dense(6, input_dim = 6, kernel_initializer = 'normal', activation = 'relu'))  
    mymodel.add(Dropout(dropoutRate))  
    mymodel.add(Dense(3, input_dim = 6, kernel_initializer = 'normal', activation = 'relu'))  
    mymodel.add(Dropout(dropoutRate))  
    mymodel.add(Dense(1, activation = 'sigmoid'))  
  
    # Compiling the model  
    my_Adam = Adam(learning_rate = learnRate)  
    mymodel.compile(loss = 'binary_crossentropy', optimizer = my_Adam, metrics = ['accuracy'])  
    return mymodel  

Explanation:

The following snippet of code has described a function as create_my_model() accepting two parameters, i.e., learnRate and dropoutRate, respectively. We have then created the model as mymodel the usage of the Sequential() function. We have additionally used the add() along with the Dense() and Dropout() function. We have then compiled the model using the compile() function.

As a result, when we execute the code, this will lead to loading the dataset, preprocessing it, and creating a machine studying model. Since we are solely involved in grasp the working of Grid Search, we haven’t carried out the train/test split, and we had fitted the model on the complete dataset.

Now, we will recognize how Grid Search makes the programmer’s existence less difficult through optimizing the parameters in the next section.

Training the Model without Grid Search

In the snippet of code proven below, we will create a model with the assist of parameter values that we decided on randomly or based totally on our instinct and see how our model performs:

File: mygrid.py

# Declaring the values of the parameter  
dropoutRate = 0.1  
epochs = 1  
batchSize = 20  
learnRate = 0.001  
  
# Creating the model object by calling the create_my_model function we created above  
mymodel = create_my_model(learnRate, dropoutRate)  
  
# Fitting the model onto the training data  
mymodel.fit(X_stdized, Y, batch_size = batchSize, epochs = epochs, verbose = 1)  

Output:

4/4 [==============================] - 41s 14ms/step - loss: 0.9364 - accuracy: 0.0000e+00

Explanation:

In the above snippet of code, we have declared the values of the parameter, i.e., dropoutRate, epochs, batchSize, and learnRate, respectively. We have then created the mannequin object by way of calling the create_my_model() function. We have then equipped the mannequin onto the training data.

As a result, the accuracy we got is 0.0000e+00.

Optimizing Hyper-parameters using Grid Search

If we are no longer using the Grid Search method, we can without delay name the fit() feature on the model we have created above. But in order to utilize the Grid Search method, we have to ignore in few arguments to the create_my_model() function. Moreover, we have to declare the grid with a number preferences to try for each parameter. Let us perform that in exceptional parts.

First of all, we will try editing the create_my_model() function in order to accept arguments from the calling function as shown below:

File: mygrid.py

def create_my_model(learnRate, dropoutRate):  
    # Creating the model  
    mymodel = Sequential()  
    mymodel.add(Dense(6, input_dim = 6, kernel_initializer = 'normal', activation = 'relu'))  
    mymodel.add(Dropout(dropoutRate))  
    mymodel.add(Dense(3, input_dim = 6, kernel_initializer = 'normal', activation = 'relu'))  
    mymodel.add(Dropout(dropoutRate))  
    mymodel.add(Dense(1, activation = 'sigmoid'))  
  
    # Compile the model  
    myadam = Adam(learning_rate = learnRate)  
    mymodel.compile(loss = 'binary_crossentropy', optimizer = myadam, metrics = ['accuracy'])  
    return mymodel  
  
# Creating the model object  
mymodel = KerasClassifier(build_fn = create_my_model, verbose = 1)  

Explanation:

In the above snippet of code, we have made some changes to the preceding create_my_model feature and used the KerasClassifier to create the model object.

Now, let us put in force the algorithm for Grid Search and in shape the dataset on it:

File: mygrid.py

# Defining the arguments that we want to use in Grid Search along  
# with the list of values that we want to try out  
learnRate = [0.001, 0.02, 0.2]  
dropoutRate = [0.0, 0.2, 0.4]  
batchSize = [10, 20, 30]  
epochs = [1, 5, 10]  
  
# Making a dictionary of the grid search parameters  
paramgrid = dict(learnRate = learnRate, dropoutRate = dropoutRate, batch_size = batchSize, epochs = epochs )  
  
# Building and fitting the GridSearchCV  
mygrid = GridSearchCV(estimator = mymodel, param_grid = paramgrid, cv = KFold(random_state = None), verbose = 10)  
  
gridresults = mygrid.fit(X_stdized, Y)  
  
# Summarizing the results in a readable format  
print("Best: " + gridresults.best_score_ + " using " + gridresults.best_params_)  
  
means = gridresults.cv_results_['mean_test_score']  
stds = gridresults.cv_results_['std_test_score']  
params = gridresults.cv_results_['params']  
  
for mean, stdev, param in zip(means, stds, params):  
    print(mean + "(" + stdev + ")" + " with: " + param)  

Output:

Best: 0.00347268912077, using {batch_size=10, dropoutRate=0.4, epochs=5, learnRate=0.2}

Explanation:

The above output shows the parameter combination which yields the quality accuracy.

At last, we can conclude that the Grid Search is quite easy to implement in the Python programming language and saved us a lot of time in human labor. We can listing down all the arguments we wanted to tune, declare the values that need to be tested, execute the code, and neglect about it. The manner is so convenient and convenient that it requires much less input from the programmer’s side. Once the fine argument combination has been found, we can utilize that for the last model.