DSL Design Patterns — Deep Dive

The Expression Problem in DSL Design

Every DSL faces the expression problem: how do you make it easy to add both new operations and new data types? Python’s dynamic dispatch and protocols offer elegant solutions.

Consider a rule engine DSL. You need rules that combine with & and |, evaluate against data, and serialize to storage:

from abc import ABC, abstractmethod
from typing import Any

class Rule(ABC):
    @abstractmethod
    def evaluate(self, context: dict) -> bool: ...

    @abstractmethod
    def serialize(self) -> dict: ...

    def __and__(self, other):
        return AndRule(self, other)

    def __or__(self, other):
        return OrRule(self, other)

    def __invert__(self):
        return NotRule(self)

class FieldRule(Rule):
    def __init__(self, field: str, op: str, value: Any):
        self.field = field
        self.op = op
        self.value = value

    def evaluate(self, context):
        actual = context.get(self.field)
        ops = {'eq': lambda a, b: a == b, 'gt': lambda a, b: a > b,
               'lt': lambda a, b: a < b, 'contains': lambda a, b: b in a}
        return ops[self.op](actual, self.value)

    def serialize(self):
        return {'type': 'field', 'field': self.field, 'op': self.op, 'value': self.value}

class AndRule(Rule):
    def __init__(self, left, right):
        self.left = left
        self.right = right

    def evaluate(self, context):
        return self.left.evaluate(context) and self.right.evaluate(context)

    def serialize(self):
        return {'type': 'and', 'left': self.left.serialize(), 'right': self.right.serialize()}

Usage reads naturally:

class F:
    def __init__(self, name):
        self.name = name
    def __gt__(self, value):
        return FieldRule(self.name, 'gt', value)
    def __eq__(self, value):
        return FieldRule(self.name, 'eq', value)

age, status = F('age'), F('status')
rule = (age > 18) & (status == 'active')
rule.evaluate({'age': 25, 'status': 'active'})  # True

Adding new rule types (e.g., ExistsRule, RegexRule) requires no changes to existing code. Adding new operations (e.g., to_sql()) requires adding a method to all rule classes — but a visitor pattern or functools.singledispatch can handle this cleanly.

Lazy Evaluation and Expression Trees

Production DSLs rarely execute immediately. Instead, they build expression trees that can be analyzed, optimized, and then executed:

class Expr:
    """Base for lazy expression tree nodes."""
    def __add__(self, other):
        return BinaryExpr('+', self, _wrap(other))

    def __mul__(self, other):
        return BinaryExpr('*', self, _wrap(other))

    def __getattr__(self, name):
        return AttrExpr(self, name)

class Column(Expr):
    def __init__(self, name):
        self._name = name

    def to_sql(self):
        return self._name

class BinaryExpr(Expr):
    def __init__(self, op, left, right):
        self.op = op
        self.left = left
        self.right = right

    def to_sql(self):
        return f"({self.left.to_sql()} {self.op} {self.right.to_sql()})"

class Literal(Expr):
    def __init__(self, value):
        self.value = value

    def to_sql(self):
        return repr(self.value)

def _wrap(value):
    return value if isinstance(value, Expr) else Literal(value)

# Usage
price = Column('price')
quantity = Column('quantity')
total = price * quantity + 10
print(total.to_sql())  # ((price * quantity) + 10)

This pattern powers SQLAlchemy’s expression language, Polars’ lazy API, and PySpark’s query plan construction. The key insight: the DSL builds a data structure (the expression tree), not a result.

Descriptor-Based DSL Fields

Python descriptors enable DSL fields that validate, transform, or track access:

class TypedField:
    def __init__(self, type_, default=None, validators=None):
        self.type = type_
        self.default = default
        self.validators = validators or []

    def __set_name__(self, owner, name):
        self.name = name
        self.private_name = f'_field_{name}'

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self  # class-level access returns descriptor
        return getattr(obj, self.private_name, self.default)

    def __set__(self, obj, value):
        if not isinstance(value, self.type):
            raise TypeError(f"{self.name} must be {self.type.__name__}, got {type(value).__name__}")
        for validator in self.validators:
            validator(value)
        setattr(obj, self.private_name, value)

class Config:
    host = TypedField(str, default='localhost')
    port = TypedField(int, default=8080, validators=[lambda v: v > 0])
    debug = TypedField(bool, default=False)

The __set_name__ hook (added in Python 3.6) is critical — it lets the descriptor know its own attribute name without requiring explicit registration.

Error Reporting in DSLs

Poor error messages are the fastest way to make a DSL unusable. Production DSLs need source tracking:

import inspect

class DSLError(Exception):
    def __init__(self, message, source_info=None):
        self.source_info = source_info
        super().__init__(self._format(message))

    def _format(self, message):
        if self.source_info:
            return f"{self.source_info['file']}:{self.source_info['line']} - {message}"
        return message

def tracked_rule(func):
    """Decorator that captures source location for error reporting."""
    frame = inspect.stack()[1]
    source_info = {'file': frame.filename, 'line': frame.lineno}

    def wrapper(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except Exception as e:
            raise DSLError(str(e), source_info) from e
    wrapper._source_info = source_info
    return wrapper

Libraries like Pydantic and attrs demonstrate the value of clear error messages that reference the user’s code, not the library internals.

Composability Patterns

Pipeline Composition

class Pipeline:
    def __init__(self, steps=None):
        self._steps = steps or []

    def __or__(self, other):
        """Compose with | operator."""
        if isinstance(other, Pipeline):
            return Pipeline(self._steps + other._steps)
        return Pipeline(self._steps + [other])

    def __call__(self, data):
        result = data
        for step in self._steps:
            result = step(result)
        return result

# Define steps as functions or callables
normalize = Pipeline([str.lower, str.strip])
tokenize = Pipeline([lambda s: s.split()])

# Compose pipelines
process = normalize | tokenize
process("  Hello World  ")  # ['hello', 'world']

Builder with Validation

class SchemaBuilder:
    def __init__(self):
        self._fields = {}
        self._validators = []

    def field(self, name, type_, **kwargs):
        self._fields[name] = {'type': type_, **kwargs}
        return self

    def validate(self, func):
        self._validators.append(func)
        return self

    def build(self):
        # Validate schema consistency before building
        for name, spec in self._fields.items():
            if spec.get('references') and spec['references'] not in self._fields:
                raise DSLError(f"Field '{name}' references unknown field '{spec['references']}'")
        return Schema(self._fields, self._validators)

Case Studies

SQLAlchemy’s Expression Language

SQLAlchemy’s Core DSL is one of the most sophisticated internal DSLs in Python. It combines:

• Operator overloading for SQL expressions (column > 5)
• Lazy expression trees compiled to SQL at execution time
• Dialect-specific compilation (PostgreSQL, MySQL, SQLite each get different SQL)
• Composable constructs (selects, joins, subqueries combine naturally)

The key design decision: expressions are never strings. They are always objects that can be inspected, transformed, and compiled. This enables optimizations and cross-database portability.

Click’s CLI DSL

Click uses decorators to build command-line interfaces:

• @click.command() registers a function as a CLI command
• @click.option() adds flags with type conversion and validation
• @click.group() creates command hierarchies

Click’s insight: compose decorators to build incrementally richer CLI definitions without ever leaving Python’s decorator syntax.

Hypothesis Property-Based Testing

Hypothesis’s strategy DSL uses operator overloading and method chaining:

from hypothesis import strategies as st
strategy = st.integers(min_value=0) | st.text(min_size=1)
complex_strategy = st.lists(strategy, min_size=1).map(sorted)

The | builds union strategies, .map() transforms outputs, and .filter() constrains them — all lazy, all composable.

Anti-Patterns

1. Magic without discoverability — if users cannot find available methods through autocomplete or docs, the DSL fails
2. Overloading common operators for non-obvious semantics — >> for function composition is fine in Haskell but confusing in Python
3. Mixing execution and construction — expression trees should be built first, executed separately
4. No escape hatch — always provide a way to drop down to raw Python/SQL/etc. when the DSL cannot express something
5. Implicit global state — DSLs that depend on import-time side effects become untestable

Testing DSLs

def test_rule_evaluation():
    rule = (F('age') > 18) & (F('status') == 'active')
    assert rule.evaluate({'age': 25, 'status': 'active'}) is True
    assert rule.evaluate({'age': 15, 'status': 'active'}) is False

def test_rule_serialization_roundtrip():
    rule = (F('age') > 18) & (F('status') == 'active')
    serialized = rule.serialize()
    restored = Rule.deserialize(serialized)
    assert restored.evaluate({'age': 25, 'status': 'active'}) is True

def test_dsl_error_messages():
    with pytest.raises(TypeError, match="port must be int"):
        config = Config()
        config.port = "not a number"

One thing to remember: The hallmark of a well-designed Python DSL is that it builds inspectable data structures (expression trees, rule objects) rather than executing immediately — enabling optimization, serialization, and cross-target compilation while keeping the user-facing API clean and composable.