𝗧𝗵𝗲 𝗔𝗿𝘁 𝗼𝗳 𝗥𝗲𝗱𝘂𝗰𝗶𝗻𝗴 𝗗𝗶𝗺𝗲𝗻𝘀𝗶𝗼𝗻𝘀
You're at an art museum, standing in front of a wall filled with vibrant, abstract paintings. Each one has so many colors, shapes, and brushstrokes — it's hard to tell what the painting is really about.
So, you squint. You back up a few steps. Suddenly, things look clearer. You can spot the patterns. Some paintings have strong lines, others are color-driven, and some are just chaos.
What you've done is a bit like PCA — you've reduced complexity to reveal structure. You didn't throw away the art, you just looked at it from a new angle to find what really matters.
PCA geometrically project a data set onto fewer dimensions, where the new variables are called principal components. This is done in such a way that the principal components are orthogonal and have the largest possible variances.
PCA is scale sensitive. For example, we might have age, height, weight and salary in a census data set. We need to look into whether we should use kilogram or pound for weights, how to normalize the salaries.
That requires feature scaling and normalization, extra care is required. Blindly standardizing all features might distort the data and make variation due to noise looks significant, twisting the calculation of principal components.
The basis of PCA is the covariance matrix and, in practice, there are two approaches to identify the principal components:
- Calculate the eigenvectors of the covariance matrix.
- Calculate the Singular Value Decomposition of the covariance matrix.
Normalization also guarantees that principal components are:
- Linearly independent. Each principal component is a linear combination that is not made of other principal components.
- Orthogonal. Meaning all principal components make a 90 degree angle with each other.