Quantitative Finance with Python — Deep Dive

Architecture of a quant research stack

Professional quant teams separate their codebase into layers that mirror the workflow: data, alpha, risk, portfolio construction, and execution. Each layer has clear inputs and outputs, enabling independent testing and parallel development.

data/          → market data loaders, alternative data parsers
alpha/         → signal generators, feature pipelines
risk/          → VaR engines, stress testing, Greeks computation
portfolio/     → optimizers, constraint engines, rebalancing logic
execution/     → order management, smart order routing, fill simulation

This separation matters because a change in data frequency (daily to minute bars) should not force a rewrite of the optimizer, and a new alpha signal should slot in without touching the execution layer.

Working with market data at scale

Efficient storage

CSV files break down past a few gigabytes. Production teams use columnar formats:

import pandas as pd

# Write to Parquet with compression
df.to_parquet("prices.parquet", engine="pyarrow", compression="zstd")

# Read only the columns you need
prices = pd.read_parquet(
    "prices.parquet",
    columns=["date", "close", "volume"],
    filters=[("date", ">=", "2024-01-01")],
)

For tick-level data, Arctic (by Man Group) or QuestDB provide time-series-optimized storage with Python bindings. DuckDB is increasingly popular for ad-hoc analytical queries directly on Parquet files without loading everything into memory.

Handling corporate actions

Stock splits, dividends, and mergers corrupt naive price series. Always use adjusted close prices for return calculations. Libraries like yfinance provide adjusted data, but verify against a paid source (Refinitiv, Bloomberg) for production research.

Building alpha signals

An alpha signal is any feature that predicts future returns with statistical significance. The research process:

1. Hypothesize a relationship (e.g., “stocks with accelerating revenue growth outperform”).
2. Construct the feature from available data.
3. Test correlation with forward returns across multiple time horizons.
4. Check for survivorship bias, look-ahead bias, and overfitting.

import numpy as np
import pandas as pd

def momentum_signal(prices: pd.DataFrame, lookback: int = 60) -> pd.Series:
    """12-month momentum minus the most recent month (classic 12-1 momentum)."""
    ret_12m = prices["close"].pct_change(252)
    ret_1m = prices["close"].pct_change(21)
    return ret_12m - ret_1m

def rank_normalize(signal: pd.Series) -> pd.Series:
    """Cross-sectional rank normalization to [-1, 1]."""
    ranked = signal.rank(pct=True)
    return 2 * ranked - 1

Avoiding look-ahead bias

The most common research mistake is using information that was not available at decision time. Strict rules:

• Use shift(1) on signals before comparing with forward returns.
• Date-stamp every data point by its availability date, not its reference date.
• Earnings data should use the filing date, not the fiscal quarter end.

Portfolio optimization

Mean-variance optimization

Markowitz mean-variance optimization finds weights that maximize return for a given risk level:

import numpy as np
from scipy.optimize import minimize

def optimize_portfolio(
    expected_returns: np.ndarray,
    cov_matrix: np.ndarray,
    risk_aversion: float = 1.0,
) -> np.ndarray:
    n = len(expected_returns)
    
    def objective(weights):
        port_return = weights @ expected_returns
        port_variance = weights @ cov_matrix @ weights
        return -(port_return - risk_aversion * port_variance)
    
    constraints = [{"type": "eq", "fun": lambda w: np.sum(w) - 1.0}]
    bounds = [(0.0, 0.15) for _ in range(n)]  # max 15% per asset
    
    result = minimize(
        objective,
        x0=np.ones(n) / n,
        method="SLSQP",
        bounds=bounds,
        constraints=constraints,
    )
    return result.x

Practical problems with mean-variance

Raw mean-variance is notoriously sensitive to estimation error — small changes in expected returns produce wildly different portfolios. Practitioners address this with:

• Shrinkage estimators for the covariance matrix (Ledoit-Wolf shrinkage via sklearn.covariance).
• Black-Litterman model to blend market equilibrium with subjective views.
• Risk parity allocation, which equalizes risk contribution rather than optimizing returns.
• Regularization (L2 penalty on weight deviations from a benchmark).

Risk modeling in practice

Value at Risk implementations

Three common approaches, each with tradeoffs:

import numpy as np
from scipy.stats import norm

def historical_var(returns: np.ndarray, confidence: float = 0.95) -> float:
    """Non-parametric: actual distribution, no normality assumption."""
    return -np.percentile(returns, (1 - confidence) * 100)

def parametric_var(returns: np.ndarray, confidence: float = 0.95) -> float:
    """Assumes normal distribution — fast but underestimates tail risk."""
    mu = np.mean(returns)
    sigma = np.std(returns)
    return -(mu + norm.ppf(1 - confidence) * sigma)

def conditional_var(returns: np.ndarray, confidence: float = 0.95) -> float:
    """Expected Shortfall: average loss beyond VaR threshold."""
    var = historical_var(returns, confidence)
    return -np.mean(returns[returns <= -var])

Historical VaR is the most common in practice because it captures fat tails that parametric methods miss. Conditional VaR (Expected Shortfall) is preferred by regulators under Basel III because it accounts for the severity of tail losses, not just their frequency.

Stress testing and scenario analysis

Beyond statistical measures, teams simulate named scenarios:

• 2008 Financial Crisis replay: apply actual 2008 daily returns to the current portfolio.
• Rate shock: model a 200 basis point overnight rate increase.
• Liquidity crisis: assume 50% wider bid-ask spreads and 30% reduced volume.

These scenarios use the same portfolio math but substitute historical or hypothetical return vectors.

Greeks and derivative pricing

For options-heavy portfolios, Python with QuantLib provides production-grade pricing:

import numpy as np
from scipy.stats import norm

def black_scholes_call(S, K, T, r, sigma):
    """European call option price via Black-Scholes."""
    d1 = (np.log(S / K) + (r + 0.5 * sigma**2) * T) / (sigma * np.sqrt(T))
    d2 = d1 - sigma * np.sqrt(T)
    return S * norm.cdf(d1) - K * np.exp(-r * T) * norm.cdf(d2)

def delta(S, K, T, r, sigma):
    d1 = (np.log(S / K) + (r + 0.5 * sigma**2) * T) / (sigma * np.sqrt(T))
    return norm.cdf(d1)

def gamma(S, K, T, r, sigma):
    d1 = (np.log(S / K) + (r + 0.5 * sigma**2) * T) / (sigma * np.sqrt(T))
    return norm.pdf(d1) / (S * sigma * np.sqrt(T))

In production, QuantLib handles the complexity of American options, exotic payoffs, and term structure bootstrapping that simple Black-Scholes cannot cover.

Execution and market microstructure

Transaction cost modeling

Backtests that ignore transaction costs are fantasies. A realistic cost model includes:

• Commission: fixed per trade or per share (increasingly zero for US equities).
• Spread cost: half the bid-ask spread per side.
• Market impact: price moves against you as your order fills. Commonly modeled as proportional to sigma * sqrt(volume_fraction).
• Slippage: delay between signal generation and execution.

def estimate_transaction_cost(
    trade_value: float,
    spread_bps: float = 5.0,
    impact_bps: float = 3.0,
    commission_bps: float = 1.0,
) -> float:
    """Total cost estimate in basis points."""
    total_bps = spread_bps / 2 + impact_bps + commission_bps
    return trade_value * total_bps / 10_000

VWAP and TWAP execution

Large orders are split over time to minimize impact:

• VWAP (Volume-Weighted Average Price): distribute order size proportionally to historical volume profile.
• TWAP (Time-Weighted Average Price): distribute evenly across time slices.

Python connects to broker APIs (Interactive Brokers via ib_insync, Alpaca, or proprietary FIX gateways) to implement these algorithms.

Performance tradeoffs

Pure Python is too slow for tick-level simulations. Common acceleration strategies:

Approach	Speedup	Complexity
Vectorized NumPy	10–100×	Low
Numba JIT	50–500×	Medium
Cython	100–1000×	High
C++ extension	500–5000×	Very high

Most teams start with NumPy vectorization and only reach for Numba when inner-loop performance becomes a bottleneck. The rule of thumb: if your backtest takes more than a few minutes, profile before optimizing.

Real-world deployment considerations

• Reproducibility: pin all library versions, seed random number generators, version-control data snapshots.
• Live vs. backtest parity: the same code path should run in both modes. Use an abstraction layer for data and order submission.
• Monitoring: track strategy P&L, position drift from targets, and data feed latency with Prometheus or Grafana dashboards.
• Regulatory compliance: MiFID II, SEC rules, and exchange-specific regulations require audit trails of every order and decision rationale.

The one thing to remember: Production quant finance in Python demands disciplined architecture — separate data, alpha, risk, and execution layers — and relentless attention to biases, costs, and reproducibility that backtests love to hide.