Commonly used regression algorithms and their characteristics in machine learning applications

Regression is one of the most powerful tools in statistics. Machine learning supervised learning algorithms are divided into two types: classification algorithms and regression algorithms. The regression algorithm is used for continuous distribution prediction and can predict continuous data rather than just discrete category labels.

Regression analysis is widely used in the field of machine learning, such as predicting product sales, traffic flow, housing prices, weather conditions, etc.

The regression algorithm is a commonly used machine learning algorithm for establishing automatic The relationship between variable X and dependent variable Y. From a machine learning perspective, it is used to build an algorithm model (function) to achieve the mapping relationship between attribute X and label Y. During the learning process, the algorithm tries to find the best parameter relationship so that the degree of fit is the best

In the regression algorithm, the final result of the algorithm (function) is a continuous data value. The input value (attribute value) is a d-dimensional attribute/numeric vector

Some commonly used regression algorithms include linear regression, polynomial regression, decision tree regression, Ridge regression, Lasso regression, ElasticNet regression, etc.

This article will introduce some common regression algorithms and their respective characteristics

• Linear regression
• Polynomial regression
• Support vector machine regression
• Decision tree regression
• Random Forest Regression
• LASSO Regression
• Ridge Returns
• ElasticNet Returns
• XGBoost Returns
• Local weighted linear regression

1. Linear regression

Linear regression It's often the first algorithm people learn about machine learning and data science. Linear regression is a linear model that assumes a linear relationship between an input variable (X) and a single output variable (y). Generally speaking, there are two situations:

Univariate linear regression is a modeling method used to analyze the relationship between a single input variable (i.e., a single feature variable) and a single output variable The relationship

Multivariable linear regression (also called multiple linear regression): It models the relationship between multiple input variables (multiple feature variables) and a single output variable .

A few key points about linear regression:

• Quick and easy to model
• It is particularly useful when the relationship you want to model is not very complex and you don't have a lot of data.
• Very intuitive understanding and explanation.
• It is very sensitive to outliers.

2. Polynomial regression

Polynomial regression is one of the most popular choices when we want to create a model for nonlinear separable data. It is similar to linear regression but uses the relationship between variables X and y to find the best way to draw a curve that fits the data points.

Several key points about polynomial regression:

• can model nonlinear separable data; linear regression cannot make it happen. In general, it is more flexible and can model some fairly complex relationships.
• Full control over the modeling of feature variables (exponents to set).
• Requires careful design. Some data knowledge is required to select the best index.
• If the index is not chosen properly, it is easy to overfit.

3. Support vector machine regression

Support vector machines are well known in classification problems. The use of SVM in regression is called Support Vector Regression (SVR). Scikit-learn has this method built into SVR().

A few key points about support vector regression:

• It is robust to outliers sex, and effective in high-dimensional spaces
• It has excellent generalization capabilities (the ability to correctly adapt to new, previously unseen data)
• If the number of features is much larger than the number of samples, it is easy to overfit

4. Decision tree regression

The decision tree is a type of classification used Non-parametric supervised learning methods for and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from data features. A tree can be viewed as a piecewise constant approximation.

A few key points about decision trees:

• Easy to understand and explain. Trees can be visualized.
• Applies to both categorical and continuous values
• The cost of using DT (i.e. predictive data) is proportional to the number of data points used to train the tree Logarithmic
• The prediction of the decision tree is neither smooth nor continuous (as shown in the figure above, it is a piecewise constant approximation)

5. Random Forest Regression

Random Forest Regression is basically very similar to Decision Tree Regression. It is a meta-estimator that can fit multiple decision trees on various subsamples of the data set and average them to improve prediction accuracy and control overfitting

random Forest regressors may perform better or worse than decision trees in regression problems (although they are generally better in classification problems) due to subtle overfitting and underfitting inherent in the tree construction algorithm. Trade-offs

A few points about random forest regression:

• Reducing overfitting in decision trees and improve accuracy.
• It also works for categorical and continuous values.
• Requires a lot of computing power and resources because it fits many decision trees to combine their outputs.

6. LASSO regression

LASSO regression is a variant of shrinkage linear regression. Shrinking is the process of shrinking data values ​​to a center point as an average. This type of regression is ideal for models with severe multicollinearity (high correlation between features)

A few points about Lasso regression:

• #It is most commonly used to eliminate automatic variables and select features.
• It is well suited for models that show severe multicollinearity (features are highly correlated with each other).
• LASSO regression utilizes L1 regularization
• LASSO regression is considered better than Ridge because it only selects some features and combines other features coefficient is reduced to zero.

7. Ridge regression

Ridge regression is very similar to LASSO regression because both techniques use shrinkage methods. Both Ridge and LASSO regression are well suited for models with severe multicollinearity problems (i.e. high correlation between features). The main difference between them is that Ridge uses L2 regularization, which means that none of the coefficients will go to zero (but close to zero) like in LASSO regression

A few points about ridge regression:

• It is very suitable for showing severe multicollinearity (features are highly correlated with each other) Model.
• Ridge regression uses L2 regularization. Features that contribute less will have coefficients close to zero.
• Ridge regression is considered worse than LASSO due to the nature of L2 regularization.

8. ElasticNet Regression

ElasticNet is another linear regression model trained using L1 and L2 regularization. It is a hybrid of Lasso and Ridge regression techniques, so it is also well suited for models that show severe multicollinearity (features are highly correlated with each other).

When weighing between Lasso and Ridge, a practical advantage is that Elastic-Net can inherit some of Ridge’s stability under rotation

9. XGBoost Regression

XGBoost is an efficient and effective implementation of the gradient boosting algorithm. Gradient boosting is a type of ensemble machine learning algorithm that can be used for classification or regression problems

XGBoost is an open source library originally developed by Chen Tianqi in his 2016 paper "XGBoost: A Scalable Tree" Developed in "Boosting System". The algorithm is designed to be computationally efficient and efficient

A few points about XGBoost:

• ##XGBoost does not perform well on sparse and unstructured data.
• The algorithm is designed to be computationally efficient and efficient, but the training time is still quite long for large data sets.
• It is sensitive to outliers.

10. Local Weighted Linear Regression

In Local Weights Linear Regression (Local Weights Linear Regression), we are also performing linear regression. However, unlike ordinary linear regression, locally weighted linear regression is a local linear regression method. By introducing weights (kernel functions), when making predictions, only some samples that are close to the test points are used to calculate the regression coefficients. Ordinary linear regression is global linear regression, which uses all samples to calculate the regression coefficient

Advantages, Disadvantages & Applicable Scenarios

##The advantage is to prevent under-fitting through kernel function weighting, but the disadvantage is also obvious that K needs to be debugged. When multiple linear regression is overfitting, you can try Gaussian kernel local weighting to prevent overfitting.

11. Bayesian Ridge Regression

The linear regression model solved using the Bayesian inference method is called Bayesian linear regression

Bayesian linear regression is a method that treats the parameters of a linear model as random variables and calculates the posterior through the prior. Bayesian linear regression can be solved by numerical methods, and under certain conditions, posterior or related statistics in analytical form can also be obtained

Bayesian linear regression has a Bayesian statistical model The basic properties of it can solve the probability density function of weight coefficients, conduct online learning and model hypothesis testing based on Bayes factor (Bayes factor)

Advantages and Disadvantages&Applicable Scenarios

#The advantage of Bayesian regression is that it has the ability to adapt to data, reuse data and prevent overfitting. In the estimation process, regularization terms can be introduced. For example, by introducing the L2 regularization term in Bayesian linear regression, Bayesian ridge regression can be realized.

The disadvantage is the learning process overhead Too big. When the number of features is less than 10, you can try Bayesian regression.

The above is the detailed content of Commonly used regression algorithms and their characteristics in machine learning applications. For more information, please follow other related articles on the PHP Chinese website!