Python Log Rotation Management — Core Concepts

Why logs need rotation

A Python web application handling 100 requests per second, logging one line per request, generates roughly 500 MB of log data per day. Without rotation, that is 15 GB per month and 180 GB per year — enough to fill most server disks.

Log rotation solves this by:

1. Splitting log files when they reach a size or time threshold
2. Keeping a fixed number of old log files
3. Deleting or compressing files beyond the retention limit

Python’s built-in rotation handlers

RotatingFileHandler — size-based rotation

import logging
from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler(
    'app.log',
    maxBytes=10 * 1024 * 1024,  # 10 MB
    backupCount=5,               # Keep 5 old files
)
handler.setFormatter(logging.Formatter(
    '%(asctime)s %(levelname)s %(name)s: %(message)s'
))

logger = logging.getLogger('myapp')
logger.addHandler(handler)
logger.setLevel(logging.INFO)

When app.log hits 10 MB, the handler renames it to app.log.1, moves app.log.1 to app.log.2, and so on. app.log.5 is deleted. A fresh app.log is created.

TimedRotatingFileHandler — time-based rotation

from logging.handlers import TimedRotatingFileHandler

handler = TimedRotatingFileHandler(
    'app.log',
    when='midnight',     # Rotate at midnight
    interval=1,          # Every 1 day
    backupCount=30,      # Keep 30 days
    utc=True,            # Use UTC for timing
)
handler.suffix = "%Y-%m-%d"  # Filename suffix for rotated files

This creates files like app.log.2026-03-27, app.log.2026-03-26, and so on. Other when options include 'H' (hourly), 'W0' (every Monday), and 'S' (every second, useful for testing).

How rotation works internally

When the handler detects a rotation trigger (file size exceeded or time threshold crossed), it:

1. Closes the current file
2. Renames existing backup files (shifting numbers up)
3. Renames the current file to the first backup slot
4. Opens a new file with the original name
5. Deletes the oldest backup if backupCount is exceeded

This happens synchronously on the logging call that triggers rotation. In high-throughput applications, this brief pause is usually negligible, but it is worth knowing about.

Combining size and time rotation

Python’s standard library does not offer a handler that rotates by both size and time. A common approach is to use TimedRotatingFileHandler for daily rotation and rely on an external tool for size-based cleanup, or use a third-party library like concurrent-log-handler.

Integration with logrotate

On production Linux servers, the system tool logrotate is often a better choice than Python-level rotation. Your Python app writes to a plain file, and logrotate handles rotation externally:

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

The copytruncate directive is important for Python: it copies the log file and then truncates the original in place, so Python’s file handle remains valid. Without it, Python would keep writing to the renamed (old) file.

An alternative is using WatchedFileHandler:

from logging.handlers import WatchedFileHandler

handler = WatchedFileHandler('app.log')

This handler detects when the underlying file has been renamed or deleted (by logrotate) and automatically reopens it. This is more efficient than copytruncate because it avoids the copy step.

Common misconception

Many developers think Python’s RotatingFileHandler compresses old log files automatically. It does not. Old files are kept as plain text. If you need compression, use logrotate with the compress option, or implement a custom handler that gzips rotated files.

Practical guidelines

Scenario	Recommendation
Small scripts and dev work	RotatingFileHandler with 5 MB / 3 backups
Production web apps	WatchedFileHandler + system logrotate
Containerized apps (Docker)	Log to stdout, let the container runtime handle rotation
Multi-process apps	WatchedFileHandler (safe for multiple writers)
High-throughput services	Log to stdout or a socket handler; rotate externally

One thing to remember: Python gives you two built-in rotation strategies — by size and by time — but production systems often get better results by logging to a plain file and letting the OS-level logrotate tool handle the rotation, compression, and cleanup.