Python Project Layout Conventions — Deep Dive

The Mechanics Behind Layout Choices

Python’s import system is path-based. When you run import my_package, the interpreter searches sys.path entries in order. This seemingly simple mechanism is the root cause of almost every layout-related bug.

Why src Layout Prevents Import Leaks

In a flat layout, the project root is typically on sys.path (because pip install -e . or running scripts from the root directory adds it). That means import my_package resolves to the local directory — even if the package was never properly installed. Tests pass locally but the published wheel might be missing files.

The src layout sidesteps this. Because src/ is not on sys.path by default, the only way to import my_package is to install it first (even in editable mode with pip install -e .). This surfaces missing __init__.py files, forgotten package_data, and broken entry points before they reach production.

# pyproject.toml for src layout (using setuptools)
[build-system]
requires = ["setuptools>=68.0", "wheel"]
build-backend = "setuptools.build_meta"

[tool.setuptools.packages.find]
where = ["src"]

[project]
name = "my-package"
version = "1.2.0"

Editable Installs in Both Layouts

Since pip 21.3 and setuptools 64, editable installs (pip install -e .) work reliably with src layout through a .pth file or import hook. Older setups sometimes required setup.py develop, which monkey-patched sys.path. If your CI uses an older pip version, upgrade before switching to src layout.

Real-World Directory Tree

A production-grade repository typically looks like this:

my-project/
├── src/
│   └── my_package/
│       ├── __init__.py
│       ├── py.typed          # PEP 561 marker for type stubs
│       ├── cli.py            # entry point for console_scripts
│       ├── core/
│       │   ├── __init__.py
│       │   ├── engine.py
│       │   └── models.py
│       ├── adapters/
│       │   ├── __init__.py
│       │   ├── database.py
│       │   └── http_client.py
│       └── _internal/
│           └── helpers.py    # underscore = private by convention
├── tests/
│   ├── conftest.py
│   ├── unit/
│   │   ├── test_engine.py
│   │   └── test_models.py
│   └── integration/
│       └── test_database.py
├── docs/
│   ├── conf.py
│   └── index.rst
├── scripts/
│   └── seed_db.py
├── pyproject.toml
├── Makefile
├── .pre-commit-config.yaml
└── README.md

Key Decisions in This Tree

• core/ and adapters/ follow a ports-and-adapters pattern, separating business logic from I/O.
• _internal/ signals that other packages should not import from it, even though Python does not enforce this at runtime.
• py.typed tells type checkers like mypy that the package ships inline type annotations.
• conftest.py at the tests root defines shared fixtures; sub-folders can have their own for scoped fixtures.

Namespace Packages

PEP 420 and the implicit namespace package mechanism (PEP 420 was withdrawn; the feature landed via PEP 382 and evolved into the current import system) allow multiple distributions to share a top-level name. For example, an organization acme can publish acme-auth and acme-billing as separate wheels:

# Repo: acme-auth
src/
└── acme/           # NO __init__.py here
    └── auth/
        ├── __init__.py
        └── tokens.py

# Repo: acme-billing
src/
└── acme/           # NO __init__.py here
    └── billing/
        ├── __init__.py
        └── invoices.py

After installing both, import acme.auth and import acme.billing work. The critical rule: the shared acme/ directory must not contain an __init__.py, or Python treats it as a regular package and only sees whichever distribution was installed last.

Automated Scaffolding with Cookiecutter and Copier

Manually creating directories for every new project invites drift. Template tools enforce consistency:

# Cookiecutter
cookiecutter gh:audreyfeldroy/cookiecutter-pypackage

# Copier (supports template updates after project creation)
copier copy gh:your-org/python-template my-new-project

Copier’s copier update command is particularly powerful — it pulls template improvements into existing projects while preserving local changes, similar to a rebase.

What to Include in an Org Template

1. Pre-configured pyproject.toml with linting tools (ruff, mypy)
2. CI pipeline file (GitHub Actions or GitLab CI)
3. Dockerfile with multi-stage build
4. Pre-commit hooks configuration
5. Standardized Makefile targets: install, test, lint, build
6. CHANGELOG.md placeholder with format instructions

Monorepo Considerations

In a monorepo, each package gets its own src/ and pyproject.toml, but shared tooling lives at the root:

monorepo/
├── packages/
│   ├── service-a/
│   │   ├── src/
│   │   ├── tests/
│   │   └── pyproject.toml
│   └── library-b/
│       ├── src/
│       ├── tests/
│       └── pyproject.toml
├── pyproject.toml       # workspace-level tool config (ruff, mypy)
├── Makefile
└── .github/

Tools like hatch and uv support workspace-level dependency resolution, letting packages reference each other without publishing to a registry first.

Testing Your Layout

A smoke test that catches most layout bugs in CI:

#!/bin/bash
set -e
python -m venv /tmp/layout-check
source /tmp/layout-check/bin/activate
pip install .
cd /tmp  # leave the project root so local imports fail
python -c "import my_package; print(my_package.__version__)"
deactivate
rm -rf /tmp/layout-check

If this script passes, your pyproject.toml, package discovery, and __init__.py files are all correct.

Tradeoffs to Acknowledge

• Src layout adds one level of nesting, which mildly annoys developers who navigate by typing paths.
• Flat layout is fine for throwaway scripts or Jupyter-heavy data science repos where packaging is irrelevant.
• Namespace packages complicate IDE auto-import because the top-level directory lacks an __init__.py.
• Over-engineering layout for a 200-line script is a waste; conventions matter most when projects grow past a single contributor.

The one thing to remember: Your layout is an API for human navigation — invest in it early, automate it with templates, and test it the same way you test code.