1. Zero-centering

Zero-centering refers to the process of adjusting the mean of a set of data points to zero. This is done by subtracting the mean from each data point. This is done to make the data more suitable for analysis. For example, if we have a set of data points that are all positive, we can zero-center them by subtracting the mean from each data point. This will make the data more suitable for analysis.

This consists of subtracting the feature-wise mean Ex[X] from all samples:

where,
Xscaled = scaled version of X, and is the mean of the samples in X.
X = original version of X.
E[X] = mean of the samples in X

This operation, if necessary, is normally reversible, and doesn’t alter relationships either among samples or among components of the same sample. In deep learning scenarios, a zero-centered dataset allows us to exploit the symmetry of some activation functions, driving our model to faster convergence.

Why do we need our mean to be zero?

1. Remove bias: The mean of the data can be influenced by external factors, rather than being a reflection of the underlying patterns in the data. By subtracting the mean, we can remove any potential bias in the data, making it more representative of the underlying patterns.

2. Improve model performance: In some machine learning algorithms, centering the mean to zero can improve the model’s performance. This is because it can make it easier for the algorithm to learn the underlying patterns in the data, particularly if there are different features with varying scales.

3. Facilitate comparison: When comparing data from different sources, it can be helpful to center the mean to zero so that the comparison is based on relative differences, rather than absolute values.

4. Reduce multicollinearity: In statistical models, multicollinearity refers to a situation where two or more independent variables are highly correlated with each other. This can cause problems in the model, such as unstable coefficients and reduced model accuracy. By centering the mean to zero, we can reduce the correlation between variables, potentially improving the model’s performance.

How does having mean zero will help in identifying underlying patterns in the data?

Centering the mean of a set of data to zero can help identify underlying patterns in the data because it allows the model to focus on the relative differences between data points, rather than being influenced by their absolute values.

When the mean is not centered to zero, the data can be biased towards higher or lower values, depending on external factors that are not necessarily related to the underlying patterns. This can make it more difficult to detect patterns in the data, particularly if there are features with different scales or variances.

By centering the mean to zero, we can remove this potential bias and make the data more representative of the underlying patterns. This can make it easier for machine learning algorithms to learn the relationships between the features and the target variable, particularly if there are nonlinear or interactive effects.

Furthermore, centering the mean to zero can help simplify the interpretation of the model. With a zero-centered data, the model coefficients represent the effect of a unit change in the input feature, holding all other features constant. This can make it easier to understand the relative contributions of each feature to the target variable, and to identify which features are most important in predicting the target variable.

Overall, centering the mean to zero can be a helpful preprocessing step to improve the accuracy and interpretability of machine learning models, particularly in situations where there are different features with varying scales or variances.

What do you mean by relative differences and absolute values?

Relative differences refer to the difference between two values relative to their scale, while absolute values refer to the difference between two values in their original units.

For example, suppose we have two samples in a dataset: sample A has a value of 100 and sample B has a value of 200. The absolute difference between these two samples is 100 (i.e., 200 - 100 = 100). However, if we center the mean to zero, we subtract the mean of the dataset (which in this case is 150), resulting in a new value of -50 for sample A and +50 for sample B. The relative difference between the two samples is now 100%, since sample B is twice as large as sample A. This relative difference is preserved even if we apply some kind of scaling or normalization to the data.

In machine learning, using relative differences instead of absolute values can be helpful because it allows the model to be more robust to differences in scale between features. If one feature has values that are much larger than the others, it may dominate the model and prevent it from learning the patterns in the other features. By centering the mean to zero, we can remove this potential bias and allow the model to focus on the relative differences between features, rather than their absolute values.

While zero-centering is a helpful preprocessing step for many machine learning algorithms, it is not always enough to guarantee that all algorithms will behave correctly. There are a few reasons for this:

1. Nonlinear relationships: Zero-centering does not take into account any nonlinear relationships between the features and the target variable. If there are nonlinear relationships, such as interactions or higher-order effects, zero-centering may not be sufficient to capture these relationships.

2. Different scales and variances: Zero-centering does not address the issue of different scales and variances across features. If some features have much larger scales or variances than others, they may still dominate the model even after zero-centering. In such cases, additional preprocessing steps such as normalization or standardization may be necessary to ensure that all features are treated equally by the model.

3. Outliers: Zero-centering is sensitive to outliers, which can affect the mean and skew the distribution of the data. If there are outliers in the data, they can still have a large impact on the model even after zero-centering. In such cases, it may be necessary to remove or adjust the outliers before applying zero-centering.

4. Algorithm-specific requirements: Some machine learning algorithms may have specific requirements that are not met by zero-centering alone. For example, some algorithms may require the data to be non-negative, or to have a specific distribution. In such cases, additional preprocessing steps may be necessary to meet the specific requirements of the algorithm.

2. Range scaling

Range scaling is used to scale the data to a fixed range, such as [0, 1] or [-1, 1]. This is done by subtracting the minimum value from each data point, and then dividing by the range (i.e., the difference between the maximum and minimum values). This is done to make the data more suitable for analysis. For example, if we have a set of data points that are all positive, we can range scale them by subtracting the minimum value from each data point, and then dividing by the range. This will make the data more suitable for analysis.

Drawbacks of range scaling

1. Sensitivity to outliers: Range scaling can be sensitive to outliers, which can affect the scaling of the entire feature. If there are outliers in the data, they can skew the range and compress the rest of the data into a smaller range, potentially reducing the accuracy of the model.

2. Loss of information: Range scaling can result in a loss of information, as the original scale of the data is lost. This can make it difficult to interpret the results and understand the impact of each feature on the model.

3. Nonlinear relationships: Range scaling does not take into account any nonlinear relationships between the features and the target variable. If there are nonlinear relationships, such as interactions or higher-order effects, range scaling may not be sufficient to capture these relationships.

4. Different scales and variances: Range scaling does not address the issue of different scales and variances across features. If some features have much larger scales or variances than others, they may still dominate the model even after range scaling. In such cases, additional preprocessing steps such as normalization or standardization may be necessary to ensure that all features are treated equally by the model.

5. Algorithm-specific requirements: Some machine learning algorithms may have specific requirements that are not met by range scaling alone. For example, some algorithms may require the data to be non-negative, or to have a specific distribution. In such cases, additional preprocessing steps may be necessary to meet the specific requirements of the algorithm.

3. Robust scaling

Robust scaling is a preprocessing technique that is designed to be more robust to outliers than other scaling techniques. It is based on the interquartile range (IQR), which is the difference between the 75th and 25th percentiles of the data. This is done to make the data more suitable for analysis. For example, if we have a set of data points that are all positive, we can robust scale them by subtracting the minimum value from each data point, and then dividing by the range. This will make the data more suitable for analysis.

It’s important to remember that this technique is not an outlier filtering method. All the existing values, including the outliers, will be scaled. The only difference is that the outliers are excluded from the calculation of the parameters, and so their influence is reduced, or completely removed.

The robust scaling procedure is very similar to the standard one, and the transformed values are obtained using the feature-wise formula:

Robust scaling is able to produce an almost perfect normal distribution N(0, I) because the outliers are kept out of the calculations and only the central points contribute to the scaling factor.

Drawbacks of robust scaling

1. Non-linear transformation: Robust scaling uses a non-linear transformation to scale the data, which can make it harder to interpret the scaled values. This can be a problem if you need to communicate the scaled data to non-technical stakeholders.

2. Outlier sensitivity: Although robust scaling is designed to account for outliers, it can still be sensitive to extreme values. In particular, if you have a large number of outliers, the scaling may not work as well as you’d like, and you may need to consider other normalization techniques.

3. Effect on correlation structure: Robust scaling can alter the correlation structure of the data, which can be a problem if you’re trying to preserve the original relationships between variables. In particular, if you have highly correlated variables, robust scaling may change the correlations in ways that make it harder to interpret the data.

4. Impact on statistical tests: Robust scaling can also impact the results of statistical tests, since the scaling may change the distribution of the data. This can make it harder to interpret the significance of any results you obtain from the data.

We can conclude this blog with a general rule of thumb: standard scaling is normally the first choice. Range scaling can be chosen as a valid alternative when it’s necessary to project the values onto a specific range, or when it’s helpful to create sparsity. If the analysis of the dataset has highlighted the presence of outliers and the task is very sensitive to the effect of different variances, robust scaling is the best choice.

4. Interview questions

1. What is zero-centering, and why might you want to use it?

2. How does range scaling work, and what are some potential drawbacks of this technique?

3. What is robust scaling, and how is it different from other normalization techniques?

4. How does robust scaling account for outliers, and why is this important?

5. What are some potential limitations of robust scaling, and when might you want to consider using other normalization techniques?

6. Can you give an example of a situation where zero-centering or range scaling might not be appropriate, and why?

7. How do you decide which normalization technique to use in a given situation?

8. What impact can normalization have on machine learning models, and why is it important to use appropriate normalization techniques?

9. How might you evaluate the effectiveness of a normalization technique, and what metrics might you use to do so?

10. How do you handle missing values when normalizing data, and what impact can missing values have on normalization techniques?