Python Shiv Zipapp — Deep Dive

Zipapp format under the hood

Python’s zipapp format (PEP 441) is deceptively simple: a zip archive prepended with a shebang line and containing a __main__.py. When you run python app.pyz, Python treats the archive as a package and executes __main__.py.

Shiv’s __main__.py is not your application — it is a bootstrap script that:

1. Computes the extraction path from the build ID.
2. Checks if extraction already exists.
3. Extracts the zip contents if needed.
4. Patches sys.path to include the extracted site-packages.
5. Imports and calls your entry point.

# Simplified version of what Shiv's __main__.py does
import zipfile, sys, os, hashlib

archive = sys.argv[0]
build_id = "a1b2c3d4"  # embedded at build time
extract_dir = os.path.expanduser(f"~/.shiv/myapp/{build_id}")

if not os.path.exists(extract_dir):
    with zipfile.ZipFile(archive) as zf:
        zf.extractall(extract_dir)

site_pkgs = os.path.join(extract_dir, "site-packages")
sys.path.insert(0, site_pkgs)

from myapp.main import run
run()

Custom preambles

Shiv supports injecting a custom preamble — Python code that runs before extraction and before your application starts:

shiv -p '/usr/bin/env python3.11' \
     --preamble preamble.py \
     -e myapp:main \
     -o myapp.pyz myapp

The preamble file runs in the bootstrap context:

# preamble.py
import os
import sys

# Redirect extraction to a custom location
os.environ.setdefault("SHIV_ROOT", "/opt/shiv-cache")

# Set application-specific environment variables
os.environ["APP_ENV"] = "production"
os.environ["LOG_LEVEL"] = "WARNING"

# Validate Python version before extraction wastes time
if sys.version_info < (3, 11):
    print("Error: requires Python 3.11+", file=sys.stderr)
    sys.exit(1)

Preambles are powerful for environment setup, validation, and logging — things you want to happen before any dependency is loaded.

Environment variables

Shiv’s behavior is controlled through environment variables:

# Change extraction root (default: ~/.shiv)
SHIV_ROOT=/opt/shiv-cache ./myapp.pyz

# Force re-extraction (useful for debugging)
SHIV_FORCE_EXTRACT=1 ./myapp.pyz

# Set a custom entry point at runtime (overrides build-time setting)
SHIV_ENTRY_POINT="myapp.debug:main" ./myapp.pyz

# Extend Python path before Shiv's path manipulation
SHIV_PREPEND_SITEPKGS=1 ./myapp.pyz

SHIV_ROOT is the most important for production — set it to a path on a fast filesystem, ideally local SSD rather than NFS.

Cache management in production

The extraction cache grows with each deployment. Without cleanup, ~/.shiv accumulates stale directories:

# List all extractions
ls -la ~/.shiv/myapp/

# Automated cleanup script — keep only the 3 most recent builds
#!/bin/bash
APP_DIR="$HOME/.shiv/myapp"
ls -dt "$APP_DIR"/*/ | tail -n +4 | xargs rm -rf

For fleet-wide management, include cleanup in your deployment script:

#!/bin/bash
set -euo pipefail

APP="myapp"
DEPLOY_DIR="/opt/apps"
SHIV_DIR="/opt/shiv-cache/$APP"

# Deploy new version
cp "myapp-${VERSION}.pyz" "$DEPLOY_DIR/myapp.pyz"
chmod +x "$DEPLOY_DIR/myapp.pyz"

# Pre-warm extraction
"$DEPLOY_DIR/myapp.pyz" --version

# Clean old extractions (keep current + 1 for rollback)
CURRENT_BUILD=$(ls -t "$SHIV_DIR" | head -1)
PREVIOUS_BUILD=$(ls -t "$SHIV_DIR" | head -2 | tail -1)
for dir in "$SHIV_DIR"/*/; do
    base=$(basename "$dir")
    if [[ "$base" != "$CURRENT_BUILD" && "$base" != "$PREVIOUS_BUILD" ]]; then
        rm -rf "$dir"
    fi
done

Building with compiled dependencies

Shiv handles compiled extensions naturally because it extracts to disk. But the build environment matters:

# Build on the target platform (or a matching container)
docker run --rm -v $(pwd):/build python:3.11-slim bash -c '
    pip install shiv
    cd /build
    shiv -r requirements.txt -e myapp:main -o myapp.pyz
'

For reproducible builds, pin shiv itself and use a lockfile:

pip install shiv==1.0.6
pip-compile requirements.in -o requirements.txt
shiv -r requirements.txt -e myapp:main \
     --build-id "$(git rev-parse --short HEAD)" \
     -o "myapp-$(git describe --tags).pyz"

Tying the build ID to the git commit makes it trivial to trace a deployed artifact back to source.

Shiv with systemd

Deploying Shiv applications as systemd services:

# /etc/systemd/system/myapp.service
[Unit]
Description=MyApp Service
After=network.target

[Service]
Type=simple
User=appuser
Environment=SHIV_ROOT=/opt/shiv-cache
ExecStart=/opt/apps/myapp.pyz serve --port 8080
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

The key detail: set SHIV_ROOT explicitly. Running as a service user, the default ~/.shiv resolves to the service user’s home, which may be /nonexistent or / on some systems.

Embedding data files

Shiv bundles everything pip installs, but sometimes you need extra files:

# Include a data directory alongside your package
shiv -r requirements.txt \
     --site-packages ./extra-data \
     -e myapp:main \
     -o myapp.pyz

The --site-packages flag adds a directory to the bundle as if it were part of site-packages. At runtime, files appear in the extracted site-packages directory.

For finding data at runtime:

import importlib.resources
import myapp

# Python 3.9+
data_path = importlib.resources.files(myapp) / "data" / "config.yaml"
with importlib.resources.as_file(data_path) as path:
    config = load_config(path)

Limitations and workarounds

No Windows support for direct execution — Windows does not honor shebang lines. Workaround: use a .bat wrapper or rename to .pyz and associate with python.exe.

No multi-platform bundles — unlike PEX, a single Shiv file targets one platform. Workaround: build in CI with matrix strategy (Linux, macOS, etc.).

Large extraction footprint — Shiv extracts everything, even if your app only imports a fraction. Workaround: minimize requirements; split large apps into focused services.

No built-in compression — extraction directories use uncompressed files. Workaround: compress the .pyz file for transfer (it is a zip, so further compression gains are minimal).

Introspection and debugging

# See what's inside a Shiv file
python -m zipfile -l myapp.pyz | head -30

# Check the build ID without running the app
python -c "
import zipfile, json
with zipfile.ZipFile('myapp.pyz') as zf:
    env = json.loads(zf.read('environment.json'))
    print(json.dumps(env, indent=2))
"

# Run with Shiv debug output
SHIV_ROOT=/tmp/debug-shiv ./myapp.pyz
ls /tmp/debug-shiv/myapp/

The environment.json embedded in every Shiv file records the build ID, entry point, Python version constraints, and build timestamp — all useful for debugging deployment issues.

One thing to remember: Shiv’s simplicity is its greatest strength — by extracting fully to disk and staying out of Python’s import machinery, it avoids the compatibility pitfalls that plague zip-based runners, making it the most predictable way to ship single-file Python applications.