Feature Selection vs. Feature Extraction

Feature selection and feature extraction are two key techniques used in machine learning to improve model performance by handling irrelevant or redundant features. While both works on data preprocessing, feature selection uses a subset of existing features whereas feature extraction transforms data into a new feature.

• Feature selection: Involves selecting a subset of the most relevant features that are actually contributing in prediction while discarding the rest features. This helps improve reducing overfitting and increased accuracy. Common techniques include filter, wrapper and embedded methods.
• Feature extraction: Transforms existing features into a new set of features that captures better underlying patterns in data. It is useful when raw data is in high dimension or complex. Techniques like PCA, LDA and Autoencoders are used for this purpose.

Difference Feature Selection and Feature Extraction Methods

Feature selection and feature extraction methods have their own advantages and disadvantages depending on the nature of the data and the task they handle. 

When to Use Feature Selection vs Feature Extraction

Use Feature Selection When

• You want to keep the original meaning of features.
• The dataset is not extremely high-dimensional.
• You need a more interpretable model.
• You want to remove redundant or irrelevant features.
• Models like Decision Trees, Random Forest, Logistic Regression benefit from clean input features.

Use Feature Extraction When

• The dataset is very high-dimensional (e.g., images, text, sensor data).
• You need to capture underlying structure not visible in raw features.
• The raw features are correlated or noisy.
• Deep learning models require dense, informative representations (e.g., Autoencoders).
• Tasks involve dimensionality reduction, such as PCA for visualization or compression.