Portfolio Optimization with Python — Core Concepts

The problem portfolio optimization solves

Given a set of assets with different expected returns, volatilities, and correlations, find the allocation weights that maximize return for a chosen level of risk — or equivalently, minimize risk for a target return. This is the foundation of Modern Portfolio Theory (MPT), for which Harry Markowitz won the Nobel Prize in 1990.

The efficient frontier

Plot every possible portfolio on a chart with risk (volatility) on the x-axis and expected return on the y-axis. Most portfolios cluster inefficiently — you can find another mix that delivers the same return with less risk. The efficient frontier is the curve along the top edge where no further improvement is possible without increasing risk.

The point on the frontier with the highest Sharpe Ratio (return per unit of risk) is called the tangency portfolio — the optimal risky portfolio for a rational investor.

Inputs you need

1. Expected returns for each asset — often estimated from historical averages or analyst forecasts.
2. Covariance matrix — measures how assets move together. A 10-asset portfolio needs a 10×10 matrix.
3. Constraints — real portfolios have rules: no short selling, minimum position sizes, sector limits.

A basic optimization in Python

import numpy as np
from scipy.optimize import minimize

def efficient_frontier_point(
    expected_returns: np.ndarray,
    cov_matrix: np.ndarray,
    target_return: float,
) -> np.ndarray:
    n = len(expected_returns)
    
    def portfolio_volatility(weights):
        return np.sqrt(weights @ cov_matrix @ weights)
    
    constraints = [
        {"type": "eq", "fun": lambda w: np.sum(w) - 1},
        {"type": "eq", "fun": lambda w: w @ expected_returns - target_return},
    ]
    bounds = [(0, 1) for _ in range(n)]  # long-only
    
    result = minimize(
        portfolio_volatility,
        x0=np.ones(n) / n,
        method="SLSQP",
        bounds=bounds,
        constraints=constraints,
    )
    return result.x

By varying target_return from the minimum to the maximum achievable return, you trace out the full efficient frontier.

Why naive optimization fails

Mean-variance optimization is mathematically elegant but practically fragile:

• Input sensitivity: small errors in expected returns produce wildly different portfolios. Changing one asset’s expected return by 0.5% can flip it from 0% weight to the maximum allowed.
• Concentration: without constraints, the optimizer loves to pile into one or two assets.
• Estimation error: expected returns are the hardest input to estimate accurately, yet the optimizer relies on them most.

Practical fixes

Problem	Solution
Extreme weights	Add upper/lower bounds per asset
Estimation error in returns	Use the Black-Litterman model or shrinkage
Unstable covariance	Apply Ledoit-Wolf shrinkage
Concentration	Add diversification constraints or use risk parity

Risk parity — a popular alternative

Instead of optimizing for returns, risk parity equalizes each asset’s contribution to total portfolio risk. This avoids the need to estimate expected returns altogether:

def risk_parity_weights(cov_matrix: np.ndarray) -> np.ndarray:
    """Simple inverse-volatility approximation to risk parity."""
    vols = np.sqrt(np.diag(cov_matrix))
    inv_vol = 1.0 / vols
    return inv_vol / inv_vol.sum()

Bridgewater’s All Weather Fund is the most famous risk parity portfolio, managing over $100 billion with this principle.

Common misconception

Many investors think optimization guarantees the best future performance. It optimizes for estimated parameters — which are always wrong to some degree. The goal is not to find the perfect portfolio but to find one that is robust to estimation errors and performs well across plausible scenarios. Optimization is a framework for disciplined allocation, not a crystal ball.

Libraries that simplify the work

• PyPortfolioOpt: high-level API for efficient frontier, Black-Litterman, and risk parity with sensible defaults.
• cvxpy: convex optimization toolkit that handles complex constraints elegantly.
• Riskfolio-Lib: specialized library with 30+ risk measures and portfolio models.

The one thing to remember: Portfolio optimization turns the diversification intuition — “do not put all your eggs in one basket” — into a rigorous mathematical framework, but its real-world value depends on honest treatment of estimation uncertainty.