Kaggle Competition Guide: How to Rank in the Top 10% Every Time
⚡ Quick Answer
Kaggle competition guide — the systematic approach to finishing in the top 10%, from EDA and baseline models to ensembling and post-competition learning, used by Kaggle Masters.
Get more content like this on Telegram!
Daily AI tips, notes & resources — free
Advertisement
Kaggle Competition Guide: How to Rank in the Top 10% Every Time
Kaggle is the world's largest data science competition platform, where participants build models against a shared leaderboard and a private final ranking. A first competition entry commonly lands outside the top 800 of 1,200 — the tenth entry, run with a systematic process, can land in the top 10%.
The gap isn't raw skill or more compute. It's methodology. Top competitors try the right things in the right order, validate every change rigorously, and combine their best models systematically instead of guessing.
This guide codifies the workflow used by Kaggle Masters — from EDA through ensembling — as a reproducible process for any competition.
Competition Selection
Your First Competition
Learning competitions — not active, prize-driven ones — are the right place to start. They carry no time pressure and come with years of public solutions to learn from.
Learning competitions (no time pressure, extensive resources):
- Titanic: Machine Learning from Disaster
→ Binary classification basics, feature engineering
- House Prices: Advanced Regression Techniques
→ Regression, handling many features, missing values
- Digit Recognizer (MNIST)
→ First neural network / CNNChoosing Active Competitions
Match the competition's data type to skills you already have — tabular data if you know scikit-learn, images if you know PyTorch, text if you know NLP. Chasing an unfamiliar domain and a leaderboard at the same time is a losing combination.
Selection criteria:
✓ Data type you have skills for (tabular/image/text/audio)
✓ Evaluation metric you understand (AUC, RMSE, F1, MAP)
✓ Dataset size you can handle (RAM/GPU constraints)
✓ Active discussion forum (shows community engagement)
✗ Avoid: prize competitions as first entries (too competitive)
✗ Avoid: proprietary domain data with no public knowledgeThe Competition Workflow
Phase 1: Understanding the Problem (Day 1-2)
Understanding the problem means knowing the evaluation metric, the data's structure, and any leakage risk before writing a single model line. Skipping this step is the single most common reason a promising competitor stalls at a mediocre score.
# Competition setup template
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# 1. Load and inspect
train = pd.read_csv('train.csv')
test = pd.read_csv('test.csv')
sample_submission = pd.read_csv('sample_submission.csv')
print(f"Train shape: {train.shape}")
print(f"Test shape: {test.shape}")
print(f"Target distribution:\n{train['target'].value_counts()}")
print(f"\nMissing values in train:\n{train.isnull().sum()[train.isnull().sum() > 0]}")
# 2. Check for test/train distribution differences (data leakage signals)
for col in train.select_dtypes(include=[np.number]).columns:
train_mean = train[col].mean()
test_mean = test[col].mean()
if abs(train_mean - test_mean) / (train_mean + 1e-8) > 0.1:
print(f"Distribution shift in {col}: train={train_mean:.3f}, test={test_mean:.3f}")Key questions to answer before modeling:
- What is the evaluation metric and how does it penalize different errors?
- Is the test data from the same time period as training? (temporal leakage)
- What features exist and what do they mean? (read the competition description)
- What external data is allowed?
Phase 2: Exploratory Data Analysis
EDA (exploratory data analysis) means looking directly at feature distributions and their relationship to the target before modeling — like a doctor reviewing a patient's chart before prescribing anything.
# Target distribution
plt.figure(figsize=(10, 4))
train['target'].hist(bins=50)
plt.title('Target Distribution')
plt.show()
# Feature correlation with target
numeric_cols = train.select_dtypes(include=[np.number]).columns.tolist()
correlations = train[numeric_cols].corr()['target'].sort_values(ascending=False)
print("Top 10 correlated features:")
print(correlations.head(11)) # 11 because target itself is included
# Categorical features
for col in train.select_dtypes(include=['object']).columns:
print(f"\n{col}:")
print(train.groupby(col)['target'].agg(['mean', 'count']))
# Distribution plots by target class
for col in correlations.head(6).index[1:]: # Skip target
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
train[col].hist(ax=axes[0], bins=30, alpha=0.7)
axes[0].set_title(f'{col} Distribution')
train.groupby('target')[col].plot(kind='kde', ax=axes[1])
axes[1].set_title(f'{col} by Target')
axes[1].legend()
plt.show()Phase 3: Baseline Model
A baseline is a simple, fast first model that gives every later change a number to beat. Skip this and you have no way to know if an "improvement" actually improved anything.
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import LabelEncoder
from sklearn.ensemble import RandomForestClassifier
import lightgbm as lgb
import numpy as np
# Simple preprocessing
def basic_preprocess(df):
df = df.copy()
# Encode categoricals
for col in df.select_dtypes(include=['object']).columns:
le = LabelEncoder()
df[col] = le.fit_transform(df[col].fillna('missing'))
# Fill numeric missing
df = df.fillna(df.median())
return df
X = basic_preprocess(train.drop(['id', 'target'], axis=1))
y = train['target']
# 5-fold CV baseline
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# LightGBM baseline (usually good starting point for tabular)
lgb_params = {
'objective': 'binary',
'metric': 'auc',
'n_estimators': 500,
'learning_rate': 0.05,
'num_leaves': 31,
'random_state': 42,
'verbose': -1
}
baseline_scores = []
for fold, (train_idx, val_idx) in enumerate(cv.split(X, y)):
X_tr, X_val = X.iloc[train_idx], X.iloc[val_idx]
y_tr, y_val = y.iloc[train_idx], y.iloc[val_idx]
model = lgb.LGBMClassifier(**lgb_params)
model.fit(X_tr, y_tr,
eval_set=[(X_val, y_val)],
callbacks=[lgb.early_stopping(50, verbose=False)])
score = model.score(X_val, y_val)
baseline_scores.append(score)
print(f"Fold {fold+1}: {score:.4f}")
print(f"\nBaseline CV score: {np.mean(baseline_scores):.4f} ± {np.std(baseline_scores):.4f}")Document this score. Every change you make gets compared to this baseline.
Phase 4: Feature Engineering
Feature engineering — building new input columns from existing data — typically produces the single biggest score gain of any phase, larger than model choice or hyperparameter tuning combined.
def engineer_features(df, train_df=None):
df = df.copy()
# 1. Interaction features between top correlated variables
df['feature_A_times_B'] = df['feature_A'] * df['feature_B']
df['feature_A_div_B'] = df['feature_A'] / (df['feature_B'] + 1e-8)
# 2. Aggregation features (if there are group IDs)
if 'group_id' in df.columns:
group_stats = df.groupby('group_id')['feature_A'].agg(['mean', 'std', 'max', 'min'])
group_stats.columns = [f'group_{c}_feature_A' for c in group_stats.columns]
df = df.merge(group_stats, on='group_id', how='left')
# 3. Target encoding for high-cardinality categoricals
# (use cross-validation to avoid leakage)
# 4. Polynomial features for top numeric features
top_features = ['feature_A', 'feature_B', 'feature_C']
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
poly_features = poly.fit_transform(df[top_features])
return df
# Test each feature addition
# If CV improves: keep the feature
# If CV stays same or drops: remove the featurePhase 5: Hyperparameter Tuning
Hyperparameter tuning searches for the settings — tree depth, learning rate, regularization — that squeeze the most performance out of your best model. Do this after feature engineering, not before: tuning a model on weak features wastes compute optimizing the wrong thing.
import optuna
def objective(trial):
params = {
'n_estimators': trial.suggest_int('n_estimators', 100, 1000),
'num_leaves': trial.suggest_int('num_leaves', 20, 300),
'max_depth': trial.suggest_int('max_depth', 3, 12),
'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3, log=True),
'min_child_samples': trial.suggest_int('min_child_samples', 5, 100),
'reg_alpha': trial.suggest_float('reg_alpha', 1e-8, 10.0, log=True),
'reg_lambda': trial.suggest_float('reg_lambda', 1e-8, 10.0, log=True),
'feature_fraction': trial.suggest_float('feature_fraction', 0.4, 1.0),
'bagging_fraction': trial.suggest_float('bagging_fraction', 0.4, 1.0),
'objective': 'binary',
'metric': 'auc',
'verbose': -1,
'random_state': 42
}
model = lgb.LGBMClassifier(**params)
cv_scores = cross_val_score(model, X, y, cv=5, scoring='roc_auc', n_jobs=-1)
return cv_scores.mean()
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=50, timeout=3600) # 1 hour budget
print(f"Best CV AUC: {study.best_value:.4f}")
print(f"Best params: {study.best_params}")Phase 6: Ensembling
Ensembling combines predictions from multiple different models into one — a jury of models instead of a single judge, and juries make fewer systematic errors than individuals.
import numpy as np
from sklearn.linear_model import Ridge
from sklearn.model_selection import StratifiedKFold
import lightgbm as lgb
import xgboost as xgb
import catboost as cb
# Train multiple base models with out-of-fold predictions
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# OOF (Out-of-Fold) predictions for stacking
oof_lgb = np.zeros(len(X))
oof_xgb = np.zeros(len(X))
test_preds_lgb = np.zeros(len(test))
test_preds_xgb = np.zeros(len(test))
for fold, (train_idx, val_idx) in enumerate(cv.split(X, y)):
print(f"Fold {fold + 1}/5")
X_tr, X_val = X.iloc[train_idx], X.iloc[val_idx]
y_tr, y_val = y.iloc[train_idx], y.iloc[val_idx]
# LightGBM
lgb_model = lgb.LGBMClassifier(**best_lgb_params)
lgb_model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)],
callbacks=[lgb.early_stopping(50, verbose=False)])
oof_lgb[val_idx] = lgb_model.predict_proba(X_val)[:, 1]
test_preds_lgb += lgb_model.predict_proba(X_test)[:, 1] / 5
# XGBoost
xgb_model = xgb.XGBClassifier(**best_xgb_params, eval_metric='auc', verbosity=0)
xgb_model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], early_stopping_rounds=50,
verbose=False)
oof_xgb[val_idx] = xgb_model.predict_proba(X_val)[:, 1]
test_preds_xgb += xgb_model.predict_proba(X_test)[:, 1] / 5
# Evaluate individual models
from sklearn.metrics import roc_auc_score
print(f"LGB OOF AUC: {roc_auc_score(y, oof_lgb):.4f}")
print(f"XGB OOF AUC: {roc_auc_score(y, oof_xgb):.4f}")
# Simple average ensemble
oof_blend = (oof_lgb + oof_xgb) / 2
test_blend = (test_preds_lgb + test_preds_xgb) / 2
print(f"Blend OOF AUC: {roc_auc_score(y, oof_blend):.4f}")
# Stacking: train meta-model on OOF predictions
meta_features = np.column_stack([oof_lgb, oof_xgb])
meta_model = Ridge()
meta_model.fit(meta_features, y)
oof_stacked = meta_model.predict(meta_features)
print(f"Stacked OOF AUC: {roc_auc_score(y, oof_stacked):.4f}")Competition-Specific Tips
For Tabular Competitions
- LightGBM is usually the strongest baseline — start there before trying anything more exotic.
- CatBoost and XGBoost add diversity to an ensemble — different tree-splitting logic catches different errors.
- Time-series data must never leak the future into training folds — a single leaked timestamp invalidates the whole CV score.
- Feature importance plots point at what to engineer next — the model is telling you where it's finding signal.
For Image Competitions
- Transfer learning from pretrained models (EfficientNet, ViT) beats training from scratch almost every time.
- Test-time augmentation (TTA) averages predictions across augmented versions of the same image, trading inference time for accuracy.
- Multi-scale training and inference improves robustness to image size variation in the test set.
For NLP Competitions
- Pretrained language models (BERT, RoBERTa, DeBERTa) are the default starting point, not an exotic choice.
- External data is often allowed — check the competition rules before assuming it isn't.
- Pseudo-labeling on test data can add a meaningful boost when done carefully.
The Mental Model of Top Competitors
Top competitors think in experiments, not guesses:
1. Hypothesis: "Adding this feature should improve CV by X"
2. Test: Run CV, measure delta
3. Keep or discard: Based on CV delta, not intuition
4. Document: Keep a log of every experiment
They don't:
- Submit before validating locally
- Chase the public LB (often leads to overfitting to the public set)
- Add random features hoping something sticks
- Copy solutions without understanding themLeaderboard Shake-Up
A leaderboard shake-up is a large drop from the public to the private leaderboard — a warning sign that a model was tuned to the public test set rather than the underlying problem.
- Trust your CV over the public leaderboard. The public LB is itself just a sample; your cross-validation is the more reliable signal.
- Verify your CV strategy actually simulates the private test set. A CV split that doesn't match the real train/test boundary gives false confidence.
- Ration your daily submissions. Use them to answer specific questions, not to chase small leaderboard bumps.
- Pick two final submissions, not one. One optimized for the public LB, one chosen for CV stability — a hedge against a shake-up.
The Bottom Line
Kaggle competitions compress years of practical ML learning into months, because the format forces rigorous evaluation and the post-competition write-ups from top finishers are some of the best ML education available anywhere.
The path to a consistent top-10% finish: understand the problem deeply, establish a rigorous CV baseline, engineer features systematically, tune carefully, and ensemble diverse models. Every competition adds a technique that carries into the next one.
For the ML skills competitions build on, see our machine learning beginners guide, feature engineering guide, and overfitting guide.
Further Reading
- Overfitting in Machine Learning: How to Detect and Fix It
- ML Engineer Roadmap 2025: From Beginner to Hired in 12 Months
- Feature Engineering Guide: Turn Raw Data into Powerful ML Inputs
- Computer Vision Tutorial: Build an Image Classifier from Scratch
- Machine Learning for Beginners: A Honest Guide to Getting Started
- Python for Machine Learning 2026 — Your First ML Project with scikit-learn
- AI API Cost Management: How to Cut LLM Costs by 80% Without Losing Quality
- FastAPI Tutorial 2026 — Build Production-Ready REST APIs with Python
Advertisement
💬 DiscussionPowered by GitHub Discussions
Frequently Asked Questions
Software Testing Expert & Prompt Engineering
Ensures every release is bug-free through rigorous testing, and crafts high-precision prompts that power our AI-driven workflows. Abdullah Al Arman Emon leads QA and prompt engineering across AiTechWorlds.
Not sure yet? Ask AI about this article
Get an instant, unbiased AI summary of “Kaggle Competition Guide: How to Rank in the Top 10% Every Time”.
Advertisement
Related Articles
Best Machine Learning Courses in 2026: Ranked After Taking Them All
The best machine learning courses in 2026 — ranked by a practitioner who completed them. Honest assessments of Coursera, Fast.ai, Kaggle, and 7 others with cost and time required.
Computer Vision Tutorial: Build an Image Classifier from Scratch
Computer vision tutorial for beginners — build a real image classifier using CNNs and PyTorch, understand how computers see images, and learn transfer learning for production results.
Feature Engineering Guide: Turn Raw Data into Powerful ML Inputs
Feature engineering guide for machine learning — practical techniques to create, transform, and select features that improve model accuracy, with Python code examples for every method.
Machine Learning for Beginners: A Honest Guide to Getting Started
Machine learning for beginners explained honestly — what ML actually is, which skills you need first, the fastest learning path, and what to build to prove you can do it.