Hyper-parameters: RandomSeachCV and GridSearchCV in Machine Learning
Original Source Here
Hyper-parameters: RandomSeachCV and GridSearchCV in Machine Learning
Techniques to improve the accuracy of the algorithm
Hyper-parameter Tuning
In this article, we will discuss hyper-parameter tuning. When we talk about improving the accuracy of the machine and deep learning model the first thing is to come’s to our mind is the tuning parameter.
Topics to be covered
1. What is hyper-parameter tuning?
2. Why do we need hyper-parameter tuning?
3. Hyper-parameter Types
4. Techniques of hyper-parameter tuning
a. GridSearchCV
b. RandomizedSearchCV
5. Bayesian Optimization -Automate Hyper-parameter Tuning(Hyper-
optimization)
What is hyper-parameter tuning?
“Hyper-parameter tuning is choosing a set of optimal hyper-parameter for learning algorithm”. we use different input parameters for different machine learning models. These input parameters are named Hyper-parameters or hyper-parameter is a parameter whose value is used to control the learning process. Hyperparameters are the parameters that can be changed in the model to get the best-suited values. For example, we decide to choose the number of hidden layers and nodes in each layer. Model performance depends heavily on hyper-parameters. Hyper-parameter tuning is also called hyper-parameter optimization.
Different types of hyper-parameters techniques in different models
- Maximum depth of decision tree
- No of trees in random forest
- K in K-nearest neighbor
- Learning rate in gradient descent
- C and sigma in support vector machines
- The penalty in logistic regression classifier i.e. L1 or L2 regularization
- The Learning rate for training a neural network
Why do we need hyper-parameter tuning?
The main objective behind hyper-parameter tuning is to get the optimized values of all parameters used in modeling to get better accuracy. It is used to prevent over-fitting.
Over-fitting: When we train the model the accuracy of training data goes high but when we test the model on new points then accuracy goes very low this condition is called over-fitting. To prevent this we use hyper-parameter tuning.
Hyper-parameter Types
The hyper-parameters have a type of data value in which we try to fix them in a range as shown below:
Techniques of hyper-parameter tuning
GridSearchCV
This library is used to find the best hyperparameters for our model to get better accuracy. This approach is called GridSearchCV because it searches for the best set of hyper-parameter from a grid of hyper-parameter values. As mention above it is the process of performing hyper-parameter tuning to determine the optimal values for a given model. The cross-validation method is used to find the training and testing set for the target estimator(model).
GridSearchCV is beneficial when we need tuned parameters for the target model and dataset. In this method multiple parameters are tested by cross-validation and the best parameters can be extracted to apply for a predictive model. After implemented tuned parameters in the model we get accuracy and loss values so, we try to find the best parameters with more accuracy and minimum loss values.
GridSearchCV is a function that comes in Scikit-learn model selection packages
(from sklearn.model_selection import GridSearchCV)
The no of parameters used in the GridSearchCV method.
In this process, we pass predefined values for hyper-parameter to the GridSearchCV function. We make a dictionary and mention the parameters with their value.
Here is an example as shown below:
In this technique, we’ll learn how to use the sklearn’s GridSearchCV to find out the best parameters of support vector machine for the following dataset:
Tunable parameters for support vector machine: {‘c’, ‘gamma’, ‘kernel’}
Example of GridSearchCV
{ ‘C’ : 1, ‘gamma’ : 0.7, ‘kernel’ : ‘rbf’ } are the best values for hyper-parameters for an SVM model for the above dataset. The SVM estimator has different values for tuned parameters for other data set.
RandomizedSearchCV
Randomized search CV is another example for finding the best hyperparameters. This method is used to implement ’fit’ and ‘score’ methods. Apart from these two it also implements “predict”, “predict proba”, “decision function”, “transform”, and “inverse transform” if they are used. This method is differing from the Grid search CV in that it doesn’t try to get tuned on all parameters but some specified ones from the distribution. When we are tuning all parameters then we make a list and if we are choosing just one parameter then it also comes from the distribution.
In this method when we are using continuous parameters then we should choose continuous distribution also in the tuning.
The drawbacks in GridSearchCV are improved by RandomSearchCV because it works also on a finite number of hyperparameters. The best parameters are set by this search approach in a random fashion in the grid. This approach reduces the unnecessary computation complexity
RandomizedSearchCV is a function that comes in Scikit-learn model selection packages
Let’s check all the parameters used in RandomSearchCV and what are the parameters to be tuned.
Example of RandomizedSearchCV
Bayesian Optimization -Automate Hyper Parameter Tuning (Hyper-optimization)
This optimization method is used to tune the parameters automatically and uses the Bayes method to find the minimum and maximum of an objective function. In this, we find the best-suited input values to a function that can give us the lowest possible output value. This technique is useful in complex and noisy data to evaluate.
Some of the common terms used in Bayesian optimization
Samples: sample is often defined as a vector of variables with a predefined range in an n-dimensional space.
Objective function: defines loss function to minimize
Domain space: it defines the range of input values to the function
Cost: Numeric value for a sample calculated via the objective function
Library — hyperopt
This open-source library is used for large-scale optimization. This library is used to optimized the real, conditional and discrete dimensions by serial and parallel techniques. Hyper-opt uses the form of Bayesian optimization for parameter tuning from which we get the best parameter for a given model.
Some of the functions used in hyper-opt for Bayesian optimization:-
Search space: hyper-opt has different functions to specify ranges for input parameters.
hp.choice (label,options)
hp.randint(label,upper)
hp.uniform(label,low,high)
fmin: This function is used to minimizes the objective function and optimize by iterates on different estimators and their tuning parameters.
parameters :- fmin(fn,
space,
algo,
max_evals=9223372036854775807,
trials=None)
trials: it is used to keep all records about hyper-parameters, loss, and other information.
from hyperopt import Trials
trials = Trials()
Steps to follow
- Importing library and its function
- Define parameters space for optimization
- Defines a function to minimize
- Trail object is analyzing the result
Conclusion: It is an article about hyper-parameters tuning which is used for improving the accuracy of the machine learning model.
I hope you like the article. Reach me on my LinkedIn and twitter.
Recommended Articles
1. NLP — Zero to Hero with Python
2. Python Data Structures Data-types and Objects
3. Exception Handling Concepts in Python
4. Why LSTM more useful than RNN in Deep Learning?
5. Neural Networks: The Rise of Recurrent Neural Networks
6. Fully Explained Linear Regression with Python
7. Fully Explained Logistic Regression with Python
8. Differences Between concat(), merge() and join() with Python
9. Data Wrangling With Python — Part 1
10. Confusion Matrix in Machine Learning
AI/ML
Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot
via WordPress https://ramseyelbasheer.io/2021/07/26/hyper-parameters-randomseachcv-and-gridsearchcv-in-machine-learning/
