ANTLR4 for Python — Core Concepts

What Is ANTLR4?

ANTLR4 (ANother Tool for Language Recognition) is a parser generator that takes a grammar file and produces source code for a lexer and parser in your target language. For Python, it generates Python classes you can import and use directly. ANTLR4 uses an Adaptive LL(*) parsing strategy — a top-down approach that dynamically decides how far ahead to look when making parsing decisions.

The Workflow

ANTLR4 has a distinct code-generation step that separates it from pure-Python parsers:

1. Write a .g4 grammar file describing your language
2. Run the ANTLR4 tool (a Java program) to generate Python source files
3. Import the generated files in your Python project
4. Use listeners or visitors to process the parsed input

# Generate Python parser from grammar
antlr4 -Dlanguage=Python3 Calculator.g4

This produces files like CalculatorLexer.py, CalculatorParser.py, CalculatorListener.py, and CalculatorVisitor.py.

Grammar Structure

An ANTLR4 grammar file (.g4) defines both lexer and parser rules:

grammar Calculator;

// Parser rules (lowercase)
expr: expr ('+'|'-') expr    # AddSub
    | expr ('*'|'/') expr    # MulDiv
    | INT                     # Int
    | '(' expr ')'           # Parens
    ;

// Lexer rules (uppercase)
INT: [0-9]+;
WS: [ \t\r\n]+ -> skip;

The # labels (like AddSub, MulDiv) create separate listener/visitor methods for each alternative, making it easy to handle different cases without checking token types manually.

Listeners vs Visitors

ANTLR4 generates two patterns for processing parse trees:

Listeners are event-driven. ANTLR4 walks the tree for you and calls enter and exit methods as it visits each node:

from antlr4 import ParseTreeWalker
from CalculatorListener import CalculatorListener

class MyListener(CalculatorListener):
    def enterAddSub(self, ctx):
        print("Entering addition/subtraction")

    def exitAddSub(self, ctx):
        print("Leaving addition/subtraction")

walker = ParseTreeWalker()
walker.walk(MyListener(), parse_tree)

Visitors give you control over traversal. You decide when to visit children and can return values:

from CalculatorVisitor import CalculatorVisitor

class EvalVisitor(CalculatorVisitor):
    def visitAddSub(self, ctx):
        left = self.visit(ctx.expr(0))
        right = self.visit(ctx.expr(1))
        op = ctx.getChild(1).getText()
        return left + right if op == '+' else left - right

    def visitInt(self, ctx):
        return int(ctx.INT().getText())

result = EvalVisitor().visit(parse_tree)

When to use which: Listeners work well for analysis passes (collecting information, validation). Visitors work better for evaluation and transformation where you need return values.

Running the Parser

from antlr4 import CommonTokenStream, InputStream
from CalculatorLexer import CalculatorLexer
from CalculatorParser import CalculatorParser

input_stream = InputStream("3 + 4 * 2")
lexer = CalculatorLexer(input_stream)
token_stream = CommonTokenStream(lexer)
parser = CalculatorParser(token_stream)
tree = parser.expr()

The pipeline follows a clear flow: input text → lexer → token stream → parser → parse tree → listener/visitor processing.

Common Misconception

Many developers think ANTLR4 is overkill for Python projects because of the Java dependency for code generation. In practice, the Java tool only runs during development (like a compiler). At runtime, the generated Python files have no Java dependency — they use only the lightweight antlr4-python3-runtime package. You can even commit the generated files to your repository and skip the generation step entirely in CI/CD.

When to Use ANTLR4

ANTLR4 shines when you need cross-language grammar reuse (same grammar for Python, Java, and TypeScript), when the grammar is complex enough to benefit from ANTLR4’s tooling (visualization, debugging), or when you need the extensive library of community-contributed grammars at github.com/antlr/grammars-v4.

One thing to remember: ANTLR4 generates Python parsers from standalone grammar files, giving you a choice between listener (event-driven) and visitor (control-driven) patterns for processing the resulting parse trees.