Consul Service Discovery in Python — Deep Dive

Consul adoption often starts with a simple service registry and evolves into a full service mesh. Getting the most out of Consul in Python requires understanding the consistency model, efficient query patterns, and the operational boundary between Consul’s responsibilities and application logic.

Registration strategies

Self-registration vs external registration

In self-registration, the Python service registers itself on startup and deregisters on shutdown:

import consul
import atexit

c = consul.Consul()

def register():
    c.agent.service.register(
        name="order-service",
        service_id=f"order-{hostname}-{port}",
        address=hostname,
        port=port,
        tags=["v2", "primary"],
        meta={"version": "2.4.1", "region": "us-east"},
        check=consul.Check.http(f"http://{hostname}:{port}/health", interval="10s", timeout="3s"),
    )

def deregister():
    c.agent.service.deregister(f"order-{hostname}-{port}")

register()
atexit.register(deregister)

Tags and meta fields enable sophisticated routing. A load balancer can query for services tagged v2 during a canary deployment, directing only a fraction of traffic to v3.

External registration uses a sidecar or orchestrator (Nomad, Kubernetes) to manage registration. This is cleaner for containers where processes may not get a chance to run atexit hooks.

Health check design

A well-designed health endpoint checks actual readiness, not just process liveness:

@app.get("/health")
async def health():
    checks = {
        "database": await check_db_pool(),
        "cache": await check_redis(),
        "disk": check_disk_space(),
    }
    healthy = all(checks.values())
    return JSONResponse(
        content={"status": "ok" if healthy else "degraded", "checks": checks},
        status_code=200 if healthy else 503
    )

Consul marks the service as critical when the endpoint returns 503. A 429 (too busy) response sets the service to warning state — still discoverable but deprioritized by smart clients.

TTL-based checks

For services that cannot expose HTTP endpoints (batch processors, CLI tools), TTL checks work well:

# Register with TTL check
c.agent.service.register(
    name="batch-processor",
    service_id="batch-1",
    check=consul.Check.ttl("30s")
)

# Heartbeat during processing
async def processing_loop():
    while True:
        batch = await get_next_batch()
        await process(batch)
        c.agent.check.ttl_pass(f"service:batch-1")

If ttl_pass is not called within 30 seconds, Consul marks the service as failed.

Efficient discovery patterns

Blocking queries (long polling)

Instead of polling Consul repeatedly, use blocking queries that return only when something changes:

index = None

async def watch_service(name):
    global index
    while True:
        new_index, services = c.health.service(
            name, passing=True, index=index, wait="30s"
        )
        if new_index != index:
            index = new_index
            update_local_cache(services)

The index parameter creates a long-poll. Consul holds the connection open until the catalog changes or the wait timeout expires. This drastically reduces API calls and gives near-instant updates.

Prepared queries

For production routing, prepared queries offer server-side logic:

c.query.create({
    "Name": "nearest-payment",
    "Service": {
        "Service": "payment-service",
        "Tags": ["v2"],
        "Failover": {
            "NearestN": 3,
            "Datacenters": ["us-west", "eu-west"]
        },
        "OnlyPassing": True
    }
})

# Execute the query
_, result = c.query.execute("nearest-payment")

Prepared queries support nearest-datacenter failover, tag filtering, and DNS integration. The query nearest-payment.query.consul resolves via DNS, making it accessible from any language.

Consul Connect (service mesh)

Consul Connect provides mutual TLS between services. Each service gets a sidecar proxy (Envoy) that handles encryption and authorization.

From Python’s perspective, the application connects to localhost:{local_port} and the sidecar handles the rest:

# Service A talks to Service B via local proxy
response = await httpx.get("http://localhost:5001/api/orders")
# The sidecar encrypts this and routes it to Service B's sidecar

Intentions define which services can communicate:

Kind = "service-intentions"
Name = "payment-service"
Sources = [
  { Name = "order-service", Action = "allow" },
  { Name = "*", Action = "deny" }
]

This creates a zero-trust network where services must be explicitly authorized.

Multi-datacenter patterns

Consul clusters in different datacenters can be federated via WAN gossip. Python services benefit from this in two ways:

1. Cross-DC discovery: Query services in remote datacenters by appending the datacenter name: c.health.service("payment", dc="eu-west").
2. Failover: Prepared queries automatically fail over to the nearest healthy datacenter when local instances are down.

Handling network partitions

During a partition between datacenters, each DC operates independently. Services in DC1 cannot discover new services in DC2, but their local catalog remains available. Python clients should cache the last known good state:

class ConsulCache:
    def __init__(self, consul_client):
        self.client = consul_client
        self.cache = {}
    
    async def get_service(self, name):
        try:
            _, services = self.client.health.service(name, passing=True)
            self.cache[name] = services
            return services
        except Exception:
            return self.cache.get(name, [])

Key-value operations for distributed coordination

Distributed locking

Consul sessions provide distributed locks via the KV store:

# Create a session
session_id = c.session.create(ttl="15s", behavior="delete")

# Acquire lock
acquired = c.kv.put("locks/migration", b"owner-1", acquire=session_id)

if acquired:
    try:
        run_migration()
    finally:
        c.kv.put("locks/migration", b"", release=session_id)
        c.session.destroy(session_id)

The ttl ensures the lock is released if the holder crashes. The behavior="delete" cleans up the key when the session expires.

Configuration watches

Monitor configuration changes and reload without restarting:

async def watch_config(prefix):
    index = None
    while True:
        new_index, data = c.kv.get(prefix, recurse=True, index=index, wait="60s")
        if new_index != index:
            index = new_index
            config = {item["Key"]: item["Value"].decode() for item in (data or [])}
            apply_config(config)

Operational considerations

Anti-entropy

Consul agents periodically sync their local state with the servers. This means brief inconsistencies are possible — a service might be registered locally but not yet visible in the catalog. For critical paths, use consistent mode:

_, services = c.health.service("payment", passing=True, consistency="consistent")

This queries the Raft leader directly, guaranteeing the freshest data at the cost of higher latency.

Performance tuning

• Set health check intervals based on tolerance: 10s for non-critical services, 3s for latency-sensitive paths.
• Use stale consistency for high-traffic discovery queries — it reduces load on the leader and returns data that is typically less than 50ms old.
• Limit the number of blocking queries per Python process. Each open blocking query holds a connection. Use a single watcher goroutine that updates an in-memory cache.
• Tag services with version metadata to enable gradual rollouts without Consul configuration changes.

Deregistration hygiene

Orphaned services (registered but never deregistered after shutdown) accumulate. Two defenses:

1. Use deregister_critical_service_after="90s" in the check definition. Consul automatically removes services that stay critical for longer than the threshold.
2. Run a periodic sweeper that queries for critical services older than a threshold and deregisters them.

One thing to remember: Consul in production requires attention to consistency modes, health check design, and cache strategies — the initial setup is simple, but resilient service discovery depends on getting these operational details right.