Feature Engineering in Python — Deep Dive

Why Feature Engineering Dominates Model Performance

Andrew Ng famously said that applied machine learning is “basically feature engineering.” While deep learning has reduced manual feature work in domains like vision and NLP, structured/tabular data — which accounts for the majority of enterprise ML — still depends heavily on thoughtful feature construction. In Kaggle competitions on tabular data, feature engineering is consistently the single biggest differentiator between top-10 finishes and median scores.

Encoding Strategies in Depth

One-Hot Encoding with Sparse Matrices

For high-cardinality categoricals, one-hot encoding can explode memory:

from sklearn.preprocessing import OneHotEncoder
import scipy.sparse

enc = OneHotEncoder(sparse_output=True, handle_unknown="ignore")
X_encoded = enc.fit_transform(df[["city"]])
# X_encoded is a scipy sparse matrix — memory efficient

For columns with thousands of unique values (zip codes, product IDs), consider alternatives like hashing or embedding layers.

Target Encoding with Regularization

Target encoding maps each category to the mean target value but introduces leakage risk. Regularized approaches blend the category mean with the global mean:

import numpy as np

def target_encode(train, col, target, smooth=20):
    global_mean = train[target].mean()
    agg = train.groupby(col)[target].agg(["mean", "count"])
    agg["smoothed"] = (agg["count"] * agg["mean"] + smooth * global_mean) / (agg["count"] + smooth)
    return train[col].map(agg["smoothed"])

The smooth parameter controls how much weight the global mean gets. Low counts fall back toward the global average, preventing overfitting on rare categories. Use out-of-fold encoding (compute on training folds, apply to validation fold) to avoid target leakage entirely.

Frequency and Weight-of-Evidence Encoding

Frequency encoding replaces categories with their occurrence count. Weight of Evidence (WoE) uses the log-odds ratio and is popular in credit scoring. Both avoid the dimensionality explosion of one-hot encoding.

Temporal Feature Engineering

Cyclic Encoding

Hour-of-day and day-of-week are cyclic: 23:00 is close to 00:00, but numerically they are far apart. Sine-cosine encoding fixes this:

import numpy as np

df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)

Lag and Rolling Features

For time-series problems, lagged values and rolling statistics are essential:

df["sales_lag_7"] = df.groupby("store_id")["sales"].shift(7)
df["sales_roll_mean_30"] = (
    df.groupby("store_id")["sales"]
    .transform(lambda x: x.rolling(30, min_periods=1).mean())
)

Be extremely careful with leakage: rolling windows must use only past data, never future data. When building features for a prediction at time t, every input must come from time < t.

Event-Distance Features

Distance in days to the nearest holiday, pay day, or promotion event can capture behavioral shifts. Pre-compute a calendar of events and merge it with your dataset.

Automated Feature Generation

Featuretools

Featuretools performs Deep Feature Synthesis (DFS), automatically generating features from relational datasets:

import featuretools as ft

es = ft.EntitySet(id="retail")
es = es.add_dataframe(dataframe=orders, dataframe_name="orders", index="order_id")
es = es.add_dataframe(dataframe=products, dataframe_name="products", index="product_id")
es = es.add_relationship("products", "product_id", "orders", "product_id")

feature_matrix, feature_defs = ft.dfs(
    entityset=es,
    target_dataframe_name="orders",
    max_depth=2,
)

DFS generates hundreds of candidates. Prune aggressively using importance scores or mutual information to keep only what matters.

Feature Selection After Generation

Automated generation can produce thousands of columns. Use a staged selection pipeline:

1. Remove zero-variance features.
2. Remove features correlated above 0.95 with existing ones.
3. Train a gradient-boosted model and rank by feature importance.
4. Keep the top-N features that cover 95 percent of cumulative importance.

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.feature_selection import SelectFromModel

selector = SelectFromModel(
    GradientBoostingClassifier(n_estimators=100, random_state=42),
    threshold="median",
)
selector.fit(X_train, y_train)
X_selected = selector.transform(X_train)

Preventing Feature Leakage

Feature leakage is the most dangerous pitfall in feature engineering. It happens when information from the target or from the future leaks into training features.

Common Sources

• Target leakage: Using a column that is computed from the target (e.g., “was the loan repaid” when predicting loan default).
• Train-test leakage: Fitting encoders or scalers on the full dataset before splitting.
• Temporal leakage: Rolling statistics that include future data points.

Prevention Checklist

1. Always split data before any transformation.
2. Fit transformers only on the training set, apply to validation and test.
3. Use sklearn.pipeline.Pipeline to encapsulate all preprocessing.
4. For time-series, enforce a strict temporal cutoff.
5. After building features, check: “Would I have this information at prediction time in production?”

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

preprocessor = ColumnTransformer([
    ("num", StandardScaler(), numeric_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_cols),
])

pipe = Pipeline([
    ("prep", preprocessor),
    ("model", GradientBoostingClassifier()),
])

pipe.fit(X_train, y_train)  # Fits scaler/encoder only on training data
score = pipe.score(X_test, y_test)

Production Considerations

Feature Stores

In production, features need to be computed consistently for training and serving. Feature stores (Feast, Tecton, Hopsworks) manage this by providing:

• A single source of truth for feature definitions.
• Point-in-time correct joins for historical training data.
• Low-latency serving for real-time inference.

Versioning and Monitoring

Features drift over time. Monitor distributions in production and retrain when drift is detected. Track feature versions alongside model versions so you can reproduce any past result.

Real-World Example: Predicting Customer Churn

A telecom company predicting churn might start with raw columns: call duration, monthly charges, contract type, tenure. Useful engineered features include:

• Charge-per-minute: monthly charges divided by total call minutes.
• Tenure bucket: binned into 0-6 months, 6-12 months, 1-3 years, 3+ years.
• Support-call trend: rolling 3-month count of customer service calls.
• Contract-end proximity: days until current contract expires.

These four features alone can lift AUC by 5-8 percentage points compared to raw inputs, according to benchmarks on the Telco Customer Churn dataset published by IBM.

Tradeoffs

Approach	Pros	Cons
Manual engineering	Domain-informed, interpretable	Time-consuming, requires expertise
Automated (Featuretools)	Fast, discovers unexpected features	Generates noise, needs pruning
Deep learning embeddings	Learns representations end-to-end	Black box, needs large data

Key Takeaway

One thing to remember: Feature engineering is where domain knowledge meets data science — the best pipelines combine systematic automation with human insight about what should matter and why.