Machine learning essentials: How to prevent overfitting?

In fact, the essence of regularization is very simple. It is a means or operation that imposes a priori restrictions or constraints on a certain problem to achieve a specific purpose. The purpose of using regularization in an algorithm is to prevent the model from overfitting. When it comes to regularization, many students may immediately think of the commonly used L1 norm and L2 norm. Before summarizing, let's first take a look at what the LP norm is?

LP norm

The norm can be simply understood as used to represent the distance in the vector space, and the definition of distance is very abstract. It can be called as long as it satisfies non-negative, reflexive, and triangle inequalities. It's distance.

LP norm is not a norm, but a set of norms, which is defined as follows:

##The range of p is [1,∞). p is not defined as a norm in the range of (0,1) because it violates the triangle inequality.

According to the change of pp, the norm also has different changes. Borrowing a classic change diagram of P norm as follows:

The above picture represents It shows the change of the unit ball when p changes from 0 to positive infinity. The unit ball defined under the P norm is a convex set, but when 0

Then the question comes, what is the L0 norm? The L0 norm represents the number of non-zero elements in the vector, expressed as follows:

We can find the least optimal sparse features by minimizing the L0 norm. item. But unfortunately, the optimization problem of L0 norm is an NP hard problem (L0 norm is also non-convex). Therefore, in practical applications, we often perform convex relaxation of L0. It is theoretically proven that the L1 norm is the optimal convex approximation of the L0 norm, so the L1 norm is usually used instead of directly optimizing the L0 norm.

L1 norm

According to the definition of LP norm we can easily get the mathematical form of L1 norm:

Pass As can be seen from the above formula, the L1 norm is the sum of the absolute values ​​of each element of the vector, also known as the "sparse regularization operator" (Lasso regularization). So the question is, why do we want sparsification? There are many benefits to sparsification, the two most direct ones are:

Feature selection
• Interpretability

L2 norm

The L2 norm is the most familiar. It is the Euclidean distance. The formula is as follows:

The L2 norm has many names. Some people call its regression "ridge regression." "(Ridge Regression), some people also call it "Weight Decay". Using the L2 norm as a regularization term can obtain a dense solution, that is, the parameter ww corresponding to each feature is very small, close to 0 but not 0; in addition, the L2 norm as a regularization term can prevent the model from catering to the training set. Too much complexity leads to overfitting, thereby improving the generalization ability of the model.

The difference between L1 norm and L2 norm

Introduce a classic diagram of PRML to illustrate the difference between L1 and L2 norm, as shown in the following figure:

As shown in the figure above, the blue circle represents the possible solution range of the problem, and the orange circle represents the possible solution range of the regular term. The entire objective function (regular term of the original problem) has a solution if and only if the two solution ranges are tangent. It can be easily seen from the above figure that since the L2 norm solution range is a circle, the tangent point is most likely not on the coordinate axis, and since the L1 norm is a rhombus (the vertex is convex), its tangent point The tangent point is more likely to be on the coordinate axis, and the point on the coordinate axis has a characteristic that only one coordinate component is non-zero, and the other coordinate components are zero, that is, it is sparse. Therefore, there is the following conclusion: L1 norm can lead to sparse solutions, and L2 norm can lead to dense solutions.

From a Bayesian prior perspective, when training a model, it is not enough to rely solely on the current training data set. In order to achieve better generalization capabilities, it is often necessary to add prior terms, and regular terms It is equivalent to adding a priori.

• The L1 norm is equivalent to adding a Laplacean prior;
• The L2 norm is equivalent to adding a Gaussian prior.

As shown in the figure below:

Dropout

Dropout is a regularization method often used in deep learning. Its approach can be simply understood as discarding some neurons with probability p during the training process of DNNs, that is, the output of the discarded neurons is 0. Dropout can be instantiated as shown in the figure below:

## We can intuitively understand the regularization effect of Dropout from two aspects:

The operation of randomly losing neurons during each round of Dropout training is equivalent to averaging multiple DNNs, so it has the effect of voting when used for prediction.
• Reduce the complex co-adaptation between neurons. When the hidden layer neurons are randomly deleted, the fully connected network becomes sparse to a certain extent, thus effectively reducing the synergistic effects of different features. In other words, some features may rely on the joint action of hidden nodes with fixed relationships, and through Dropout, it effectively organizes the situation where some features are effective only in the presence of other features, increasing the robustness of the neural network. Great sex.

Batch Normalization

Batch Normalization is strictly a normalization method, mainly used to accelerate the convergence of the network, but it also has a certain degree of regularization effect. .

Here is a reference to the explanation of covariate shift in Dr. Wei Xiushen’s Zhihu answer.

Note: The following content is quoted from Dr. Wei Xiushen’s Zhihu answer. Everyone knows that a classic assumption in statistical machine learning is “the data distribution of source space (source domain) and target space (target domain)” (distribution) is consistent”. If they are inconsistent, then new machine learning problems arise, such as transfer learning/domain adaptation, etc. Covariate shift is a branch problem under the assumption of inconsistent distribution. It means that the conditional probabilities of the source space and the target space are consistent, but their marginal probabilities are different. If you think about it carefully, you will find that indeed, for the output of each layer of the neural network, because they have undergone intra-layer operations, their distribution is obviously different from the distribution of the input signals corresponding to each layer, and the difference will increase as the depth of the network increases. are large, but the sample labels they can "indicate" remain unchanged, which meets the definition of covariate shift.

The basic idea of ​​BN is actually quite intuitive, because the activation input value of the neural network before nonlinear transformation (X=WU B, U is the input) as the depth of the network deepens, its distribution gradually shifts or Change (ie the above-mentioned covariate shift). The reason why training converges slowly is generally because the overall distribution gradually approaches the upper and lower limits of the value range of the nonlinear function (for the Sigmoid function, it means that the activation input value X=WU B is a large negative or positive value) , so this causes the gradient of the low-level neural network to disappear during backpropagation, which is the essential reason why the convergence of deep neural networks in training becomes slower and slower. BN uses a certain standardization method to force the distribution of the input value of any neuron in each layer of the neural network back to the standard normal distribution with a mean of 0 and a variance of 1, to avoid the gradient dispersion problem caused by the activation function. So rather than saying that the role of BN is to alleviate covariate shift, it is better to say that BN can alleviate the gradient dispersion problem.

Normalization, Standardization & Regularization

We have mentioned regularization before, here we briefly mention normalization and standardization. Normalization: The goal of normalization is to find a certain mapping relationship to map the original data to the [a, b] interval. Generally, a and b will take combinations of [−1,1], [0,1]. There are generally two application scenarios:

Convert the number into a decimal between (0, 1)
• Convert the dimensional number into a dimensionless number

Commonly used min-max normalization:

Standardization (Standardization): Use the theorem of large numbers to transform the data into a standard normal distribution. The standardization formula is:

The difference between normalization and standardization:

We can explain it simply like this: normalized scaling is "flattened" to the interval (determined only by extreme values), while standardized scaling is It is more "elastic" and "dynamic" and has a lot to do with the distribution of the overall sample. Note:

• Normalization: Scaling is only related to the difference between the maximum and minimum values.
• Standardization: Scaling is related to each point and is reflected by variance. Contrast this with normalization, in which all data points contribute (through the mean and standard deviation).

Why standardization and normalization?

• Improve model accuracy: After normalization, the features between different dimensions are numerically comparable, which can greatly improve the accuracy of the classifier.
• Accelerate model convergence: After standardization, the optimization process of the optimal solution will obviously become smoother, making it easier to correctly converge to the optimal solution. As shown below:

The above is the detailed content of Machine learning essentials: How to prevent overfitting?. For more information, please follow other related articles on the PHP Chinese website!