Feature Engineering Guide: Turn Raw Data into Powerful ML Inputs
⚡ Quick Answer
Feature engineering guide for machine learning — practical techniques to create, transform, and select features that improve model accuracy, with Python code examples for every method.
Get more content like this on Telegram!
Daily AI tips, notes & resources — free
Advertisement
Feature Engineering Guide: Turn Raw Data into Powerful ML Inputs
Feature engineering is the process of shaping raw data into the input variables a model actually learns from. Weeks of tuning a neural network's architecture and hyperparameters can buy 1-2% accuracy; three well-chosen new features from a domain expert can buy 8%.
New ML practitioners consistently underrate this step. A model — even a deep, complex one — can only learn patterns present in the features you hand it. If the real signal is an interaction between two variables, a ratio of three, or a temporal aggregation, the model cannot invent that on its own; you have to compute it and hand it over.
This guide covers every major feature engineering technique with Python code, when to apply each, and how to think about feature creation from a domain perspective.
Numerical Features
Scaling and Normalization
Scaling rewrites numeric features onto a common range so no single feature dominates just because its raw values happen to be larger.
Algorithms like SVM, neural networks, logistic regression, and KNN are sensitive to magnitude — a "salary" column in the tens of thousands will swamp an "age" column in the tens, even if age matters just as much.
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler
# Generate example data
data = pd.DataFrame({
'age': [25, 45, 35, 55, 28],
'salary': [50000, 120000, 75000, 180000, 45000],
'years_experience': [2, 20, 10, 30, 3]
})
# StandardScaler: mean=0, std=1 — best for most algorithms
standard = StandardScaler()
data_standard = pd.DataFrame(
standard.fit_transform(data),
columns=data.columns
)
# MinMaxScaler: scales to [0, 1] — good for neural networks
minmax = MinMaxScaler()
data_minmax = pd.DataFrame(
minmax.fit_transform(data),
columns=data.columns
)
# RobustScaler: uses median and IQR — best when outliers are present
robust = RobustScaler()
data_robust = pd.DataFrame(
robust.fit_transform(data),
columns=data.columns
)
print("StandardScaler:\n", data_standard.round(3))| Scaler | Best for |
|---|---|
| StandardScaler | Default choice — works well when features are roughly normally distributed |
| MinMaxScaler | Fixed-range inputs, e.g. neural networks that expect values in [0, 1] |
| RobustScaler | Data with significant outliers — uses median and IQR instead of mean and variance |
Transformations for Skewed Features
A right-skewed distribution has a long tail of high values — income, price, and count data all look this way, with most values clustered low and a few outliers stretching far to the right.
import matplotlib.pyplot as plt
from scipy import stats
# Salary is right-skewed: most people earn moderate amounts, few earn very high
salary = np.array([30000, 35000, 40000, 45000, 50000, 60000, 75000, 100000, 250000, 500000])
# Log transformation (compress large values)
salary_log = np.log1p(salary) # log(x+1) to handle zeros
# Box-Cox transformation (finds optimal transformation)
salary_boxcox, lambda_val = stats.boxcox(salary)
print(f"Optimal Box-Cox lambda: {lambda_val:.3f}")
# Visualize before/after
fig, axes = plt.subplots(1, 3, figsize=(15, 4))
axes[0].hist(salary, bins=20)
axes[0].set_title('Original (right-skewed)')
axes[1].hist(salary_log, bins=20)
axes[1].set_title('Log transformed')
axes[2].hist(salary_boxcox, bins=20)
axes[2].set_title('Box-Cox transformed')
plt.tight_layout()
plt.show()Binning Continuous Variables
Binning groups a continuous variable into ranges — turning exact age into "Young Adult," "Middle," "Senior" — which can expose non-linear relationships a raw number hides.
df = pd.DataFrame({'age': [22, 25, 31, 42, 48, 55, 63, 70]})
# Equal-width bins
df['age_group'] = pd.cut(df['age'],
bins=[0, 30, 45, 60, 100],
labels=['Young Adult', 'Middle', 'Senior', 'Elder'])
# Quantile-based bins (equal-size groups)
df['age_quantile'] = pd.qcut(df['age'], q=4, labels=['Q1', 'Q2', 'Q3', 'Q4'])
print(df[['age', 'age_group', 'age_quantile']])Categorical Features
One-Hot Encoding
One-hot encoding turns a category with N possible values into N binary columns — "city" with values New York, London, Tokyo becomes three yes/no columns, since a model can't do arithmetic on the string "Tokyo."
from sklearn.preprocessing import OneHotEncoder
import pandas as pd
df = pd.DataFrame({
'city': ['New York', 'London', 'Tokyo', 'New York', 'Paris', 'London'],
'category': ['A', 'B', 'A', 'C', 'B', 'A']
})
# pandas get_dummies (simple, good for exploration)
dummies = pd.get_dummies(df, columns=['city', 'category'], drop_first=True)
# scikit-learn OneHotEncoder (better for pipelines)
encoder = OneHotEncoder(drop='first', sparse_output=False, handle_unknown='ignore')
encoded = encoder.fit_transform(df[['city', 'category']])
print("Encoded shape:", encoded.shape)
print("Feature names:", encoder.get_feature_names_out())Ordinal Encoding
Ordinal encoding maps ordered categories to increasing numbers — High School, Bachelor, Master, PhD become 0, 1, 2, 3, preserving the real-world ranking that one-hot encoding would throw away.
from sklearn.preprocessing import OrdinalEncoder
df = pd.DataFrame({'education': ['High School', 'Bachelor', 'Master', 'PhD', 'Bachelor']})
encoder = OrdinalEncoder(categories=[['High School', 'Bachelor', 'Master', 'PhD']])
df['education_encoded'] = encoder.fit_transform(df[['education']])
# High School=0, Bachelor=1, Master=2, PhD=3
print(df)Target Encoding (for High-Cardinality)
Target encoding replaces each category with the average target value for that category — useful once a column has hundreds of distinct values and one-hot encoding would explode into hundreds of columns.
import numpy as np
import pandas as pd
from sklearn.model_selection import KFold
def target_encode_cv(train_df, val_df, column, target, n_splits=5):
"""Cross-validated target encoding (prevents data leakage)"""
# Train set: encode using cross-validation to avoid leakage
kf = KFold(n_splits=n_splits, shuffle=True, random_state=42)
train_encoded = np.zeros(len(train_df))
for train_idx, val_idx in kf.split(train_df):
mean_map = train_df.iloc[train_idx].groupby(column)[target].mean()
train_encoded[val_idx] = train_df.iloc[val_idx][column].map(mean_map)
# Fill unmapped with global mean
global_mean = train_df[target].mean()
train_encoded = np.where(np.isnan(train_encoded), global_mean, train_encoded)
# Validation/test set: encode using all training data
mean_map_full = train_df.groupby(column)[target].mean()
val_encoded = val_df[column].map(mean_map_full).fillna(global_mean)
return train_encoded, val_encoded.values
# Example usage
train_df = pd.DataFrame({
'city': ['NYC', 'LA', 'NYC', 'SF', 'LA', 'NYC', 'SF', 'LA'],
'target': [1, 0, 1, 1, 0, 0, 1, 1]
})Datetime Features
A single timestamp hides several useful signals at once — hour of day, day of week, weekend or not — and splitting them out gives the model each one directly instead of forcing it to reverse-engineer them from a raw date.
import pandas as pd
df = pd.DataFrame({
'transaction_time': pd.to_datetime([
'2024-01-15 09:23:00', '2024-07-04 18:45:00',
'2024-12-24 14:30:00', '2024-03-17 02:15:00'
])
})
# Extract all useful time components
df['hour'] = df['transaction_time'].dt.hour
df['day_of_week'] = df['transaction_time'].dt.dayofweek # 0=Monday, 6=Sunday
df['day_of_month'] = df['transaction_time'].dt.day
df['month'] = df['transaction_time'].dt.month
df['quarter'] = df['transaction_time'].dt.quarter
df['year'] = df['transaction_time'].dt.year
df['is_weekend'] = df['day_of_week'].isin([5, 6]).astype(int)
df['is_business_hours'] = ((df['hour'] >= 9) & (df['hour'] < 17)).astype(int)
# Cyclical encoding (hour 23 is close to hour 0 — linear encoding misses this)
df['hour_sin'] = np.sin(2 * np.pi * df['hour'] / 24)
df['hour_cos'] = np.cos(2 * np.pi * df['hour'] / 24)
df['month_sin'] = np.sin(2 * np.pi * df['month'] / 12)
df['month_cos'] = np.cos(2 * np.pi * df['month'] / 12)
print(df[['transaction_time', 'hour', 'is_weekend', 'is_business_hours', 'hour_sin', 'hour_cos']])Cyclical encoding matters because clocks wrap around: hours 23 and 0 are one hour apart, but plain integers put them at opposite ends of the scale. Sine/cosine encoding represents that circular distance correctly.
Interaction Features
An interaction feature combines two or more columns into one that captures a relationship neither column shows alone — income divided by age reveals earning power relative to career stage, something neither raw number states directly.
df = pd.DataFrame({
'age': [25, 45, 35, 55, 28],
'income': [50000, 120000, 75000, 180000, 45000],
'credit_score': [650, 780, 720, 800, 600]
})
# Ratio features (often more informative than raw values)
df['income_per_age'] = df['income'] / df['age']
df['credit_score_normalized'] = df['credit_score'] / 850 # Normalize to max possible
# Polynomial features (capture non-linear relationships)
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
poly_features = poly.fit_transform(df[['age', 'income', 'credit_score']])
poly_df = pd.DataFrame(poly_features, columns=poly.get_feature_names_out())
print("Polynomial feature names:", poly.get_feature_names_out())Aggregation Features
Aggregation rolls up many rows into one summary row — dozens of a customer's individual transactions become a handful of features like total spend and average order size, sized correctly for a model that expects one row per customer.
transactions = pd.DataFrame({
'customer_id': [1, 1, 1, 2, 2, 3, 3, 3, 3],
'amount': [50, 120, 75, 200, 45, 30, 80, 150, 25],
'category': ['food', 'retail', 'food', 'retail', 'food', 'food', 'retail', 'retail', 'food'],
'date': pd.to_datetime(['2024-01-01', '2024-01-05', '2024-01-10',
'2024-01-03', '2024-01-08', '2024-01-02',
'2024-01-04', '2024-01-09', '2024-01-12'])
})
# Aggregate to customer level
customer_features = transactions.groupby('customer_id').agg(
total_transactions=('amount', 'count'),
total_spend=('amount', 'sum'),
avg_spend=('amount', 'mean'),
max_spend=('amount', 'max'),
min_spend=('amount', 'min'),
spend_std=('amount', 'std'),
unique_categories=('category', 'nunique'),
days_active=('date', lambda x: (x.max() - x.min()).days)
).reset_index()
# Category-specific features
category_pivot = transactions.pivot_table(
values='amount', index='customer_id', columns='category', aggfunc='sum', fill_value=0
)
category_pivot.columns = [f'spend_{c}' for c in category_pivot.columns]
customer_features = customer_features.merge(category_pivot, on='customer_id')
print(customer_features)Handling Missing Values
Missing data needs a strategy, not a reflex — the right fix depends on why the value is absent, and picking the wrong one quietly injects false signal into the model.
from sklearn.impute import SimpleImputer, KNNImputer
import pandas as pd
import numpy as np
df = pd.DataFrame({
'age': [25, np.nan, 35, 45, np.nan],
'income': [50000, 80000, np.nan, 120000, 45000],
'education': ['Bachelor', 'Master', np.nan, 'PhD', 'Bachelor']
})
# Strategy 1: Simple imputation
# Numerical — use median (robust to outliers)
num_imputer = SimpleImputer(strategy='median')
df[['age', 'income']] = num_imputer.fit_transform(df[['age', 'income']])
# Categorical — use most frequent
cat_imputer = SimpleImputer(strategy='most_frequent')
df[['education']] = cat_imputer.fit_transform(df[['education']])
# Strategy 2: Add missingness indicator (when missing is informative)
df_with_indicator = pd.DataFrame({
'age': [25, np.nan, 35, 45, np.nan],
'income': [50000, 80000, np.nan, 120000, 45000]
})
# First, create indicator columns
df_with_indicator['age_missing'] = df_with_indicator['age'].isnull().astype(int)
df_with_indicator['income_missing'] = df_with_indicator['income'].isnull().astype(int)
# Then impute
df_with_indicator[['age', 'income']] = SimpleImputer(strategy='median').fit_transform(
df_with_indicator[['age', 'income']]
)
# Strategy 3: KNN imputation (uses similar rows to impute)
knn_imputer = KNNImputer(n_neighbors=3)
df_knn = pd.DataFrame({
'age': [25, np.nan, 35, 45, np.nan],
'income': [50000, 80000, np.nan, 120000, 45000]
})
df_knn_imputed = pd.DataFrame(knn_imputer.fit_transform(df_knn), columns=df_knn.columns)Feature Selection
Feature selection removes columns that add noise instead of signal — trimming a dataset from 1,000 loosely relevant features down to the 20 that actually move the model's predictions.
from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif
from sklearn.feature_selection import RFE
from sklearn.ensemble import RandomForestClassifier
import matplotlib.pyplot as plt
# Method 1: Feature importance from tree models
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)
feature_importance = pd.Series(rf.feature_importances_, index=feature_names).sort_values(ascending=False)
print("Top 10 features by importance:")
print(feature_importance.head(10))
# Method 2: Statistical selection (SelectKBest)
selector = SelectKBest(score_func=mutual_info_classif, k=10)
X_selected = selector.fit_transform(X_train, y_train)
selected_features = [feature_names[i] for i in selector.get_support(indices=True)]
print("Selected features:", selected_features)
# Method 3: Remove highly correlated features
def remove_correlated_features(df, threshold=0.95):
corr_matrix = df.corr().abs()
upper_triangle = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool))
to_drop = [col for col in upper_triangle.columns if any(upper_triangle[col] > threshold)]
print(f"Removing {len(to_drop)} highly correlated features: {to_drop}")
return df.drop(columns=to_drop)
X_train_uncorrelated = remove_correlated_features(pd.DataFrame(X_train, columns=feature_names))The Feature Engineering Workflow
The steps below run in order, but the last two loop — you circle back to selection every time a new feature is added.
1. Explore raw data
- distributions, missing values, outliers
- correlation with target
- domain-specific patterns
2. Basic preprocessing
- Handle missing values
- Encode categoricals
- Scale numerics
3. Create new features
- Datetime decomposition
- Interaction terms
- Domain-specific aggregations
- Ratio features
4. Select features
- Remove highly correlated
- Feature importance ranking
- Validate: do added features improve CV score?
5. Iterate
- Test each feature's contribution
- Remove features that don't improve validation scoreWhere Domain Knowledge Meets ML Skill
Feature engineering is where the two intersect. The best features come from understanding the problem deeply — why customers churn, what makes a transaction fraudulent, what time patterns drive sales — and no automated feature-generation tool replaces that understanding.
The practical rule: test every feature on validation data before shipping it to production. A feature that looks informative can still add noise instead of signal; cross-validation scores, not intuition, tell you which is true.
For the modeling skills that use these features, see our scikit-learn tutorial and machine learning beginners guide.
Further Reading
- Scikit-Learn Tutorial: Build Your First ML Model in 30 Minutes
- Neural Networks Explained: From Perceptron to Deep Learning
- Supervised vs Unsupervised Learning: The Complete Comparison
- Machine Learning for Beginners: A Honest Guide to Getting Started
- Kaggle Competition Guide: How to Rank in the Top 10% Every Time
- LLM Context Window Explained: Why It Matters and How to Use It
- GPT-4 vs Claude vs Gemini: Which AI Model Is Best in 2025?
- The Python Libraries Every Developer Must Know in 2025
Advertisement
💬 DiscussionPowered by GitHub Discussions
Frequently Asked Questions
Software Testing Expert & Prompt Engineering
Ensures every release is bug-free through rigorous testing, and crafts high-precision prompts that power our AI-driven workflows. Abdullah Al Arman Emon leads QA and prompt engineering across AiTechWorlds.
Not sure yet? Ask AI about this article
Get an instant, unbiased AI summary of “Feature Engineering Guide: Turn Raw Data into Powerful ML Inputs”.
Advertisement
Related Articles
Best Machine Learning Courses in 2026: Ranked After Taking Them All
The best machine learning courses in 2026 — ranked by a practitioner who completed them. Honest assessments of Coursera, Fast.ai, Kaggle, and 7 others with cost and time required.
Computer Vision Tutorial: Build an Image Classifier from Scratch
Computer vision tutorial for beginners — build a real image classifier using CNNs and PyTorch, understand how computers see images, and learn transfer learning for production results.
Kaggle Competition Guide: How to Rank in the Top 10% Every Time
Kaggle competition guide — the systematic approach to finishing in the top 10%, from EDA and baseline models to ensembling and post-competition learning, used by Kaggle Masters.
Machine Learning for Beginners: A Honest Guide to Getting Started
Machine learning for beginners explained honestly — what ML actually is, which skills you need first, the fastest learning path, and what to build to prove you can do it.