Overfitting in Machine Learning: How to Detect and Fix It
⚡ Quick Answer
Overfitting explained — how to detect it with learning curves, fix it with regularization, dropout, and cross-validation, and build ML models that generalize to new data.
Get more content like this on Telegram!
Daily AI tips, notes & resources — free
Advertisement
Overfitting in Machine Learning: How to Detect and Fix It
Overfitting happens when a model learns the noise in its training data instead of the pattern behind it. The result: near-perfect training accuracy and mediocre real-world performance.
Here's the scale of the trap: a classification model that hit 99% training accuracy dropped to 64% the moment it saw production data. It had memorized examples, not learned a rule.
Overfitting is the most common — and most expensive — failure mode in machine learning. High training accuracy is cheap. Performance on data the model has never seen is what matters.
This guide covers the intuition behind overfitting, how to detect it with learning curves, and the full toolkit of remedies — regularization, dropout, early stopping, and more.
The Core Intuition
Imagine trying to predict a stock's price. You have 10 years of daily prices — 2,500 data points.
Underfitting model: "The stock goes up 0.1% per year on average." Simple, but it misses all the real patterns — seasonal effects, momentum, market correlations.
Overfitting model: A 2,500-term polynomial that exactly fits every data point in the training set. It "explains" the training data perfectly but predicts complete nonsense for tomorrow's price.
Good model: One that captures real patterns (seasonal trends, momentum) without memorizing noise (random daily fluctuations). It makes useful predictions on days it hasn't seen.
The key insight: we don't care how well the model performs on data it was trained on. We care how it performs on data it hasn't seen.
Detecting Overfitting
Learning Curves
The definitive diagnostic tool. Plot training and validation performance over training time:
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve
from sklearn.ensemble import RandomForestClassifier
import numpy as np
def plot_learning_curves(model, X, y, cv=5):
train_sizes, train_scores, val_scores = learning_curve(
model, X, y,
cv=cv,
train_sizes=np.linspace(0.1, 1.0, 10),
scoring='accuracy',
n_jobs=-1
)
train_mean = train_scores.mean(axis=1)
train_std = train_scores.std(axis=1)
val_mean = val_scores.mean(axis=1)
val_std = val_scores.std(axis=1)
plt.figure(figsize=(10, 6))
plt.plot(train_sizes, train_mean, label='Training accuracy', color='blue')
plt.fill_between(train_sizes, train_mean - train_std, train_mean + train_std, alpha=0.15, color='blue')
plt.plot(train_sizes, val_mean, label='Validation accuracy', color='orange')
plt.fill_between(train_sizes, val_mean - val_std, val_mean + val_std, alpha=0.15, color='orange')
plt.xlabel('Training set size')
plt.ylabel('Accuracy')
plt.title('Learning Curves')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()
model = RandomForestClassifier(n_estimators=100, random_state=42)
plot_learning_curves(model, X, y)Reading the learning curves comes down to one number: the gap between training and validation accuracy.
| Pattern | Training acc. | Validation acc. | Diagnosis |
|---|---|---|---|
| Large gap | 0.98 | 0.73 | Overfitting — model is too complex |
| Both low | 0.71 | 0.69 | Underfitting — model is too simple |
| Small gap, both high | 0.87 | 0.84 | Good fit |
Validation Curve
Shows how model performance changes with a key hyperparameter:
from sklearn.model_selection import validation_curve
param_range = [1, 5, 10, 20, 50, 100, 200]
train_scores, val_scores = validation_curve(
RandomForestClassifier(random_state=42),
X, y,
param_name='n_estimators',
param_range=param_range,
cv=5,
scoring='accuracy'
)
plt.plot(param_range, train_scores.mean(axis=1), label='Train')
plt.plot(param_range, val_scores.mean(axis=1), label='Validation')
plt.xlabel('n_estimators')
plt.ylabel('Accuracy')
plt.legend()
plt.show()Remedies
1. Regularization
Regularization adds a penalty to the loss function for large weights, discouraging the model from fitting noise:
L2 Regularization (Ridge / Weight Decay):
# scikit-learn (uses 'C' = 1/lambda, so smaller C = more regularization)
from sklearn.linear_model import LogisticRegression, Ridge
lr_model = LogisticRegression(C=0.1, random_state=42) # Stronger regularization
ridge = Ridge(alpha=1.0) # alpha = lambda (higher = more regularization)
# PyTorch: weight_decay parameter in optimizer
optimizer = optim.Adam(model.parameters(), lr=0.001, weight_decay=1e-4)L1 Regularization (Lasso):
from sklearn.linear_model import Lasso, LogisticRegression
lasso = Lasso(alpha=0.01) # Higher alpha = more regularization, more sparsity
lr_l1 = LogisticRegression(penalty='l1', C=0.1, solver='liblinear')Elastic Net (combines L1 and L2):
from sklearn.linear_model import ElasticNet
elastic = ElasticNet(alpha=0.1, l1_ratio=0.5) # l1_ratio: balance between L1 and L22. Dropout (Neural Networks)
Dropout randomly disables neurons during training, preventing co-adaptation:
import torch.nn as nn
class RegularizedNet(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
super().__init__()
self.network = nn.Sequential(
nn.Linear(input_size, hidden_size),
nn.ReLU(),
nn.Dropout(p=0.5), # Randomly zero 50% of neurons during training
nn.Linear(hidden_size, hidden_size),
nn.ReLU(),
nn.Dropout(p=0.3), # Less aggressive in later layers
nn.Linear(hidden_size, output_size)
)
def forward(self, x):
return self.network(x)Key: Dropout is only active during training. model.train() enables it; model.eval() disables it (all neurons active, predictions are deterministic).
3. Early Stopping
Stop training when validation performance stops improving:
class EarlyStopping:
def __init__(self, patience=10, min_delta=1e-4):
self.patience = patience
self.min_delta = min_delta
self.counter = 0
self.best_loss = float('inf')
def should_stop(self, val_loss):
if val_loss < self.best_loss - self.min_delta:
self.best_loss = val_loss
self.counter = 0
return False
else:
self.counter += 1
return self.counter >= self.patience
# Usage in training loop
early_stopper = EarlyStopping(patience=15)
for epoch in range(200):
train_loss = train_one_epoch(model, train_loader)
val_loss = evaluate(model, val_loader)
if early_stopper.should_stop(val_loss):
print(f"Early stopping at epoch {epoch}")
break4. Data Augmentation
Augmentation increases effective dataset size by creating modified versions of training examples:
# Image augmentation with torchvision
from torchvision import transforms
aggressive_augmentation = transforms.Compose([
transforms.RandomHorizontalFlip(p=0.5),
transforms.RandomVerticalFlip(p=0.3),
transforms.RandomRotation(30),
transforms.RandomResizedCrop(224, scale=(0.7, 1.0)),
transforms.ColorJitter(brightness=0.3, contrast=0.3, saturation=0.3, hue=0.1),
transforms.RandomGrayscale(p=0.1),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])
# Tabular data augmentation: add Gaussian noise
import numpy as np
def augment_tabular(X, noise_scale=0.05):
noise = np.random.normal(0, noise_scale * X.std(axis=0), X.shape)
return X + noise5. Cross-Validation
Instead of a single train/test split, use k-fold cross-validation:
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.ensemble import GradientBoostingClassifier
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
model = GradientBoostingClassifier()
scores = cross_validate(
model, X, y,
cv=cv,
scoring=['accuracy', 'f1'],
return_train_score=True
)
print(f"Train Accuracy: {scores['train_accuracy'].mean():.4f} ± {scores['train_accuracy'].std():.4f}")
print(f"Val Accuracy: {scores['test_accuracy'].mean():.4f} ± {scores['test_accuracy'].std():.4f}")
print(f"Gap: {(scores['train_accuracy'].mean() - scores['test_accuracy'].mean()):.4f}")6. Reduce Model Complexity
Simpler models are less prone to overfitting:
# Random Forest: control tree depth and minimum samples
rf_constrained = RandomForestClassifier(
n_estimators=100,
max_depth=5, # Limit tree depth
min_samples_leaf=10, # Each leaf needs 10+ samples
min_samples_split=20, # Splits need 20+ samples
random_state=42
)
# Neural Network: use fewer layers and neurons
class SimpleNet(nn.Module):
def __init__(self):
super().__init__()
self.network = nn.Sequential(
nn.Linear(20, 32), # Fewer neurons
nn.ReLU(),
nn.Linear(32, 2) # No hidden layers to overfit
)7. Batch Normalization
Reduces internal covariate shift and acts as a mild regularizer:
class BNRegularizedNet(nn.Module):
def __init__(self):
super().__init__()
self.network = nn.Sequential(
nn.Linear(20, 128),
nn.BatchNorm1d(128), # Normalize layer outputs
nn.ReLU(),
nn.Dropout(0.3),
nn.Linear(128, 64),
nn.BatchNorm1d(64),
nn.ReLU(),
nn.Linear(64, 2)
)The Bias-Variance Tradeoff Visualized
The bias-variance tradeoff is the tension between a model too simple to learn the pattern (high bias) and one so complex it memorizes noise (high variance).
| High Bias (Underfitting) | Sweet Spot | High Variance (Overfitting) | |
|---|---|---|---|
| Train accuracy | 0.65 | 0.88 | 0.98 |
| Validation accuracy | 0.64 | 0.85 | 0.72 |
| Gap | 0.01 | 0.03 | 0.26 |
| Example model | Linear regression, few features | Random forest, moderate regularization | Deep neural net, no regularization |
Start simple, add complexity until validation performance stops improving, then stop. That single rule replaces most of the guesswork in model selection.
Practical Checklist
When you have overfitting:
□ Check training vs. validation accuracy gap
- Gap < 5%: acceptable
- Gap 5-15%: mild overfitting — start here
- Gap > 15%: significant overfitting — needs attention
□ Start with: more data (or data augmentation)
□ Add regularization (weight_decay, dropout)
□ Try early stopping
□ Reduce model complexity (fewer layers, shallower trees)
□ Check for data leakage (future data in training features?)
□ Use cross-validation for more reliable evaluationConclusion
Overfitting is information, not failure. A model that overfits has enough capacity to learn — it just needs constraints or more data to generalize. Plot the learning curve for every model; it's the fastest diagnostic you have.
Work through remedies in order of effort: regularization first (cheapest to add), dropout for neural networks, then more data or reduced complexity if the gap persists.
The goal was never the best training accuracy. It's the best performance on data the model has never seen.
For the broader ML workflow context, see our scikit-learn tutorial and machine learning beginners guide.
Further Reading
- Supervised vs Unsupervised Learning: The Complete Comparison
- Best Machine Learning Courses in 2025: Ranked After Taking Them All
- Feature Engineering Guide: Turn Raw Data into Powerful ML Inputs
- Scikit-Learn Tutorial: Build Your First ML Model in 30 Minutes
- ML Engineer Roadmap 2025: From Beginner to Hired in 12 Months
- AI Hallucination Explained: Why LLMs Make Things Up (and How to Fix It)
- Python Decorators and Generators — Advanced Python Made Simple 2026
- Transformer Architecture Explained: The Architecture Behind All Modern AI
Advertisement
💬 DiscussionPowered by GitHub Discussions
Frequently Asked Questions
Software Testing Expert & Prompt Engineering
Ensures every release is bug-free through rigorous testing, and crafts high-precision prompts that power our AI-driven workflows. Abdullah Al Arman Emon leads QA and prompt engineering across AiTechWorlds.
Not sure yet? Ask AI about this article
Get an instant, unbiased AI summary of “Overfitting in Machine Learning: How to Detect and Fix It”.
Advertisement
Related Articles
Best Machine Learning Courses in 2026: Ranked After Taking Them All
The best machine learning courses in 2026 — ranked by a practitioner who completed them. Honest assessments of Coursera, Fast.ai, Kaggle, and 7 others with cost and time required.
Computer Vision Tutorial: Build an Image Classifier from Scratch
Computer vision tutorial for beginners — build a real image classifier using CNNs and PyTorch, understand how computers see images, and learn transfer learning for production results.
Feature Engineering Guide: Turn Raw Data into Powerful ML Inputs
Feature engineering guide for machine learning — practical techniques to create, transform, and select features that improve model accuracy, with Python code examples for every method.
Kaggle Competition Guide: How to Rank in the Top 10% Every Time
Kaggle competition guide — the systematic approach to finishing in the top 10%, from EDA and baseline models to ensembling and post-competition learning, used by Kaggle Masters.