Python Task Cancellation — Deep Dive

Cancellation Mechanics Inside Task

The Task.cancel() method does more than set a flag. Here’s the internal flow from CPython’s tasks.py:

def cancel(self, msg=None):
    if self.done():
        return False
    self._cancel_count += 1
    self._cancel_message = msg

    if self._fut_waiter is not None:
        # Currently awaiting a Future — cancel it
        if self._fut_waiter.cancel(msg=msg):
            return True

    # Not currently awaiting — schedule __step with CancelledError
    self._must_cancel = True
    self._loop.call_soon(self.__step, CancelledError(msg))
    return True

Key detail: _cancel_count tracks how many times cancel() has been called. This is critical for the uncancel() mechanism introduced in Python 3.9.

CancelledError’s Heritage

In Python 3.8 and earlier, CancelledError was a subclass of concurrent.futures.CancelledError. Starting in Python 3.9, it became a subclass of BaseException (not Exception). This was a breaking change — a bare except Exception: no longer catches cancellation.

# Python 3.8: CancelledError is Exception
# Python 3.9+: CancelledError is BaseException

# This code behaves differently across versions:
try:
    await some_operation()
except Exception:
    # Python 3.8: catches CancelledError (bad!)
    # Python 3.9+: does NOT catch CancelledError (correct!)
    pass

This change was deliberate — cancellation should not be accidentally swallowed by generic exception handlers.

The uncancel() Method (Python 3.9+)

Task.uncancel() decrements the _cancel_count. This enables a pattern where intermediate code can temporarily suppress cancellation:

async def with_retry(coro_fn, retries=3):
    for attempt in range(retries):
        try:
            return await coro_fn()
        except asyncio.CancelledError:
            if attempt < retries - 1:
                # Suppress this cancellation for retry
                asyncio.current_task().uncancel()
                continue
            raise

The timeout() context manager in Python 3.11 uses uncancel() internally. When a timeout fires, it cancels the task. When the async with block exits, it calls uncancel() to restore the previous cancellation state — otherwise a timeout inside a larger operation would cancel the outer operation too.

async def outer():
    async with asyncio.timeout(5):
        await inner()  # If this times out...
    # ...we still reach here. timeout() uncancels after catching
    await more_work()  # This runs normally

Cancel Scopes: How timeout() Works Internally

The asyncio.Timeout class (Python 3.11+) implements a cancel scope:

class Timeout:
    def __init__(self, when):
        self._when = when
        self._task = None
        self._cancel_handle = None

    async def __aenter__(self):
        self._task = asyncio.current_task()
        if self._when is not None:
            loop = asyncio.get_running_loop()
            self._cancel_handle = loop.call_at(self._when, self._cancel)
        return self

    def _cancel(self):
        self._task.cancel(msg=_CANCEL_MSG)

    async def __aexit__(self, exc_type, exc_val, exc_tb):
        if self._cancel_handle:
            self._cancel_handle.cancel()
        if exc_type is asyncio.CancelledError and self._task.uncancel() == 0:
            raise TimeoutError from exc_val
        return False

The uncancel() return value tells you the remaining cancel count. If it’s 0, the only cancellation was from this timeout — safe to convert to TimeoutError. If it’s >0, an outer cancellation is pending — re-raise CancelledError.

Shielding Deep Dive

asyncio.shield() creates an outer Future that wraps an inner Task:

async def shield(aw):
    inner = ensure_future(aw)
    outer = loop.create_future()

    def _inner_done(inner):
        if outer.cancelled():
            return  # Outer was cancelled, but inner completed
        if inner.cancelled():
            outer.cancel()
        elif inner.exception():
            outer.set_exception(inner.exception())
        else:
            outer.set_result(inner.result())

    inner.add_done_callback(_inner_done)
    # Caller awaits outer; cancelling outer doesn't cancel inner
    return await outer

The pitfall: if the outer (shielded) future is cancelled, _inner_done sees outer.cancelled() and silently drops the result. The inner task completes successfully but nobody collects its result. This can mask exceptions. Always handle the inner task’s result explicitly if shield is cancelled:

inner_task = asyncio.create_task(critical_operation())
try:
    result = await asyncio.shield(inner_task)
except asyncio.CancelledError:
    # Inner task is still running! Wait for it:
    result = await inner_task  # Or handle differently

Building Cancellation-Safe Code

Pattern 1: Atomic Cleanup Sections

Use asyncio.shield() or Task.uncancel() for cleanup that must not be interrupted:

async def process_message(msg):
    result = await compute(msg)
    # Cleanup must complete — protect from cancellation
    try:
        await asyncio.shield(save_result(result))
    except asyncio.CancelledError:
        await save_result(result)  # Retry unshielded if shield was cancelled
        raise

Pattern 2: Cancellation-Aware Loops

async def consumer(queue):
    while True:
        try:
            item = await queue.get()
        except asyncio.CancelledError:
            # Drain remaining items before shutting down
            while not queue.empty():
                item = queue.get_nowait()
                await process_final(item)
            raise

        await process(item)

Pattern 3: Graceful Shutdown with TaskGroup

async def server():
    async with asyncio.TaskGroup() as tg:
        listener = tg.create_task(accept_connections())
        # When the TaskGroup is cancelled, all child tasks are cancelled
        # Each task gets CancelledError and can clean up

Edge Cases and Gotchas

Double Cancellation

If cancel() is called twice before the task processes the first one, _cancel_count is 2. The task needs two uncancel() calls to fully suppress cancellation. This is rare but happens with nested timeouts.

Cancellation During finally

If a task is cancelled while executing a finally block that contains an await, the CancelledError is thrown again at that await. This can interrupt cleanup:

async def risky():
    try:
        await work()
    finally:
        await slow_cleanup()  # CancelledError can land HERE too

Workaround: shield the cleanup or use synchronous cleanup when possible.

Generator-Based Coroutines

Legacy @asyncio.coroutine generators handle cancellation differently — CancelledError is thrown via generator.throw(), which has subtly different semantics for exception chaining. Another reason to use native async def.

One thing to remember: Cancellation in asyncio is a counted, cooperative protocol — cancel() increments a counter, uncancel() decrements it, and the timeout() context manager relies on this counting to distinguish its own cancellation from outer cancellations.