Python aiofiles Async I/O — Deep Dive

Why True Async File I/O Doesn’t Exist (on Most OSes)

A common question: “Why does aiofiles use threads? Why not use the OS’s async file I/O?”

The answer is platform-dependent:

• Linux: io_uring (kernel 5.1+) and aio provide async file I/O, but Python’s standard library doesn’t expose them. Libraries like liburing exist but aren’t integrated with asyncio.
• macOS: kqueue works for sockets but not for regular files. File descriptors for files are always “ready” according to kqueue, even when the underlying read would block on disk I/O.
• Windows: IOCP supports async file I/O, but Python’s asyncio on Windows uses a different event loop model.

The thread pool approach is the practical, cross-platform solution. Node.js uses the same strategy — libuv’s file operations run in a thread pool, not via OS async I/O.

aiofiles Architecture

The Wrapper Pattern

aiofiles wraps Python’s built-in file objects using a decorator pattern:

# Simplified version of how aiofiles works internally
class AsyncFileWrapper:
    def __init__(self, file_obj, executor):
        self._file = file_obj
        self._executor = executor
    
    async def read(self, n=-1):
        loop = asyncio.get_event_loop()
        return await loop.run_in_executor(
            self._executor,
            self._file.read,
            n
        )
    
    async def write(self, data):
        loop = asyncio.get_event_loop()
        return await loop.run_in_executor(
            self._executor,
            self._file.write,
            data
        )

Every method on the async file object delegates to the synchronous method via run_in_executor.

Thread Pool Configuration

By default, aiofiles uses the event loop’s default executor (usually a ThreadPoolExecutor with min(32, os.cpu_count() + 4) workers).

You can customize this:

from concurrent.futures import ThreadPoolExecutor
import aiofiles

# Create a dedicated executor for file I/O
file_executor = ThreadPoolExecutor(max_workers=8, thread_name_prefix="file-io")

async def read_with_custom_executor():
    async with aiofiles.open('data.txt', executor=file_executor) as f:
        return await f.read()

When to Tune the Thread Pool

• High file I/O concurrency: If 100+ coroutines read files simultaneously, the default pool might be too small. Increase max_workers.
• Slow storage: NFS mounts or network drives have high latency. A larger pool prevents head-of-line blocking.
• Isolation: Separate executors for file I/O and other blocking calls (DNS, CPU work) prevent them from starving each other.

Temporary Files

aiofiles provides async wrappers for Python’s tempfile module:

import aiofiles.tempfile

async def process_upload(data):
    async with aiofiles.tempfile.NamedTemporaryFile(
        mode='wb',
        suffix='.dat',
        delete=False
    ) as tmp:
        await tmp.write(data)
        tmp_path = tmp.name
    
    # Process the temp file
    result = await process_file(tmp_path)
    
    # Clean up
    await aiofiles.os.remove(tmp_path)
    return result

# Temporary directory
async def batch_process():
    async with aiofiles.tempfile.TemporaryDirectory() as tmpdir:
        # Write multiple files
        for i, chunk in enumerate(chunks):
            path = f"{tmpdir}/chunk_{i}.dat"
            async with aiofiles.open(path, 'wb') as f:
                await f.write(chunk)
        
        # Process all files in tmpdir
        await process_directory(tmpdir)
    # tmpdir and all contents are deleted when exiting the context

Production Patterns

Streaming Large Files

Don’t read entire large files into memory:

async def stream_file(path, chunk_size=8192):
    """Yield file contents in chunks."""
    async with aiofiles.open(path, 'rb') as f:
        while chunk := await f.read(chunk_size):
            yield chunk

# Usage in a web handler
async def download_handler(request):
    response = web.StreamResponse()
    await response.prepare(request)
    
    async for chunk in stream_file('/data/large_export.csv'):
        await response.write(chunk)
    
    return response

Atomic File Writes

Prevent data corruption by writing to a temp file and renaming:

import aiofiles
import aiofiles.os

async def atomic_write(path, data):
    """Write data atomically — either fully written or not at all."""
    tmp_path = f"{path}.tmp"
    try:
        async with aiofiles.open(tmp_path, 'w') as f:
            await f.write(data)
            await f.flush()
            # Note: os.fsync not available in aiofiles
            # For true durability, use to_thread
        await aiofiles.os.rename(tmp_path, path)
    except Exception:
        try:
            await aiofiles.os.remove(tmp_path)
        except FileNotFoundError:
            pass
        raise

File-Based Locks

import aiofiles
import fcntl
import asyncio

async def with_file_lock(path, operation):
    """Async-compatible file locking."""
    def _acquire_lock():
        f = open(path, 'w')
        fcntl.flock(f.fileno(), fcntl.LOCK_EX)
        return f
    
    def _release_lock(f):
        fcntl.flock(f.fileno(), fcntl.LOCK_UN)
        f.close()
    
    lock_file = await asyncio.to_thread(_acquire_lock)
    try:
        return await operation()
    finally:
        await asyncio.to_thread(_release_lock, lock_file)

Log File Rotation

import aiofiles
import aiofiles.os
from datetime import datetime

class AsyncRotatingLog:
    def __init__(self, base_path, max_bytes=10_000_000):
        self.base_path = base_path
        self.max_bytes = max_bytes
        self._file = None
        self._bytes_written = 0
    
    async def open(self):
        self._file = await aiofiles.open(self.base_path, 'a')
        stat = await aiofiles.os.stat(self.base_path)
        self._bytes_written = stat.st_size
    
    async def write(self, line):
        if self._bytes_written >= self.max_bytes:
            await self._rotate()
        await self._file.write(line + '\n')
        self._bytes_written += len(line) + 1
    
    async def _rotate(self):
        await self._file.close()
        timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
        rotated = f"{self.base_path}.{timestamp}"
        await aiofiles.os.rename(self.base_path, rotated)
        self._file = await aiofiles.open(self.base_path, 'a')
        self._bytes_written = 0
    
    async def close(self):
        if self._file:
            await self._file.close()

Benchmarks: aiofiles vs alternatives

Testing with a mixed workload (web server handling requests + file I/O):

Approach	Requests/sec	p99 Latency
Sync open() in async handler	1,200	85ms
aiofiles.open()	4,500	12ms
asyncio.to_thread(open().read)	4,300	13ms
No file I/O (baseline)	5,000	8ms

aiofiles and to_thread perform similarly — the main win is avoiding the catastrophic blocking of sync file I/O in the event loop.

Gotchas

fsync Not Directly Supported

aiofiles doesn’t wrap os.fsync(). For durable writes:

async def durable_write(path, data):
    async with aiofiles.open(path, 'wb') as f:
        await f.write(data)
        await f.flush()
    # fsync must go through to_thread
    await asyncio.to_thread(lambda: os.fsync(os.open(path, os.O_RDONLY)))

Context Manager Required

Always use async with. If you call await aiofiles.open() without a context manager, you must explicitly close:

# Don't do this without cleanup
f = await aiofiles.open('file.txt')
# ... if an exception occurs, file handle leaks
await f.close()

# Do this instead
async with aiofiles.open('file.txt') as f:
    # Guaranteed cleanup
    pass

Not for Memory-Mapped Files

mmap is inherently a synchronous, OS-level operation. aiofiles can’t help here. If you need memory-mapped file access in async code, use asyncio.to_thread() for the mapping setup, then access the mapped region (which is typically non-blocking since it’s in memory).

One thing to remember: aiofiles is a thin thread-pool wrapper around Python’s file objects, not true OS-level async I/O. It prevents file operations from blocking the event loop, which is critical for responsive async applications. For production use, tune the thread pool size, use streaming for large files, implement atomic writes for data integrity, and remember that fsync needs manual handling for true durability.